Re: [basex-talk] XSLT version
Hi, You always had to get the Saxon HE JAR and the xmlresolve JAR yourself to transform XSLT 2.0 and higher and put it in lib/custom. This is using the BaseX ZIP distribution, I don't know where this is when you use setup, I don't use anything but ZIP). By default the XSL transform built into JRE is used and that is Xerces XSLT 1.0 as far as I know. Last I tried you can get any Saxon HE you like even 12.x and it will work with BaseX 8.6.7 Best regards Omar Am 08.02.2024 um 10:50 schrieb Martin Dowling: Thank you, Liam. I sorted this out by upgrading the version of the Saxon processor in my /lib/ directory from 9.7 to 9.9. I did already have xsl:stylesheet/@version = '2.0' in my stylesheet, and BaseX appeared to be set to run XSLT 2.0: running xslt:transform() in the GUI returned a value of "2.0". However, the older Saxon version didn't handle XSLT 2.0 functions. Regards, Martin -Original Message- From: Liam R. E. Quin Sent: 08 February 2024 00:20 To: Martin Dowling;basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] XSLT version On Wed, 2024-01-10 at 17:09 +, Martin Dowling wrote: Hi I’m trying to run a query in BaseX (version 8.6.7) which calls an XSLT 2.0 stylesheet using xslt:transform(). I’m getting errors that clearly imply the system does not recognise XSLT 2.0 functions, and is expecting an XSLT 1.0 stylesheet. Did you get sorted out? You need to put a copy of your XSLT processor, e.g. Saxon, in the lib/custom directory, e.g. saxonpe.jar and of course make sure the stylesheet has version="2" on the xsl:stylesheet element. liam -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
Re: [basex-talk] RESTXQ knocked out!
Hi Hans-Juergen, This smells like you did not "uninstall" all of your last version of BaseX. I only use unzipped .zip distributions myself so I can only tell you how I upgrade those: * have all special JARs (saxon or whatever you need) in lib/custom * delete all files in lib * delete BaseX.jar * unzip the new version of basex and copy lib and BaseX.jar to your project * also replace the .bat and .sh files in bin if you did not edit them for say more RAM or different port. if you edited them you need to move these changes over. Best regards Am 01.02.2024 um 11:34 schrieb Hans-Juergen Rennau: Dear BaseX people, suddenly RESTXQ does not work any more on my machine. (Here, "suddenly" means when using RESTXQ again after a pause of three months and having installed a new BaseX version in the meantime, 11.0 beta.) Any attempt to access a RESTXQ-defined URL produces a mysterious error message: HTTP ERROR 500 java.lang.NoClassDefFoundError: org/basex/build/text/TextOptions URI:/dba STATUS:500 MESSAGE:java.lang.NoClassDefFoundError: org/basex/build/text/TextOptions SERVLET:RESTXQ CAUSED BY:java.lang.NoClassDefFoundError: org/basex/build/text/TextOptions CAUSED BY:java.lang.ClassNotFoundException: org.basex.build.text.TextOptions Caused by: java.lang.NoClassDefFoundError: org/basex/build/text/TextOptions at org.basex.http.web.WebModule.parse(WebModule.java:63) at org.basex.http.web.WebModules.parse(WebModules.java:376) at org.basex.http.web.WebModules.parse(WebModules.java:367) at org.basex.http.web.WebModules.cache(WebModules.java:336) at org.basex.http.web.WebModules.find(WebModules.java:149) at org.basex.http.web.WebModules.restxq(WebModules.java:116) Emptying the lib/custom folder did not help. Neither did a reinstallation of BaseX (see PS). With kind regards - Hans-Jürgen PS: After observing the error, I tried to install the official release (10.7). To my amazement, the version did not change, it remained the 11.0 beta snapshot, even after restarting the machine. And, of course, the error remained. (After this, I felt knocked out myself. Is there /anything/ that I understand?) -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
Re: [basex-talk] Processing stack traces in XQuery code
Hi Christian,
So I already forgot about the first few lines of the message (I always
cut them away).
What I miss is the rest, the stack trace, the list of calls that got me
to that line:
I need something like
`Stopped at { $err:module }, { $err:line-number }/{ $err:column-number }:
{ $err:description }
{ $err:stack-trace }`
I create XML and JSON according to an RFC and I want to be able to have
the trace there.
I already use { $err:module }, { $err:line-number }, {
$err:column-number } and { $err:description } E. g:
https://tools.ietf.org/html/rfc7807/tests
_:an-error: testError Test1 Test2 Test3
Test1 Test2 Test3
https://tools.ietf.org/html/rfc7807/tests/an-error
500
/app/tests/api-problem-rest-test.xqm, 38/10
- /app/tests/api-problem-rest-test.xqm, 34/16
- /app/tests/api-problem-rest-test.xqm, 30/15
- /app/tests/api-problem-rest-test.xqm, 26/15
- /app/tests/api-problem-rest-test.xqm, 21/58
- /app/api-problem/api-problem.xqm, 41/26
3.55
While in 10.7+ I only get
https://tools.ietf.org/html/rfc7807/tests
_:an-error: testError Test1 Test2 Test3
Test1 Test2 Test3
https://tools.ietf.org/html/rfc7807/tests/an-error
500
/app/tests/api-problem-rest-test.xqm, 38/10
2.15
See
https://github.com/acdh-oeaw/api-problem4restxq/blob/master/tests/api-problem-rest-test.xqm#L15-L23
for details.
There is a docker image using 10.6 for quick testing.
docker run --rm -it -p 5000:5000 ghcr.io/acdh-oeaw/api-problem4restxq vs
docker run --rm -it -p 5000:5000
ghcr.io/acdh-oeaw/api-problem4restxq:27f2a42
Look at to http://localhost:5000
Best regards
Omar
Am 12.01.2024 um 14:14 schrieb Christian Grün:
Hi Omar,
Thanks for your observation.
We have removed the BaseX-specific string output, as it was redundant.
You should be able to reproduce the old output as follows (using the
new XQuery 4 string template syntax):
`Stopped at { $err:module }, { $err:line-number }/{ $err:column-number }:
{ $err:description }`
As you’ve already discovered the responsible commit, you may have
discovered the new XQuery 4 $err:map variable: It bundles all other
error values into a single variable.
And @Martin Honnen thanks for mentioning fn:stack-trace. The function
is still under discussion and might be dropped again: The
specification gives processors much freedom in optimizing its
expressions, so it’s hardly possible to define the function in a way
that’s not completely implementation-defined.
Hope this helps,
Christian
--
Mag. Ing. Omar Siam
Austrian Center for Digital Humanities and Cultural Heritage
Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences
Stellvertretende Behindertenvertrauensperson | Deputy representative for
disabled persons
Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria
T: +43 1 51581-7295
omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
Re: [basex-talk] Processing stack traces in XQuery code
I think stack traces were removed in this commit from basex-core/src/main/java/org/basex/query/expr/Catch.java Revision: fc2376f6c9c250babfac7a7161b8a61740878b2b Author: Christian Grün Date: 17.05.2023 11:25:01 Message: [XQ4] try/catch, $err:map Modified: basex-api/src/main/java/org/basex/http/restxq/RestXqFunction.java Modified: basex-core/src/main/java/org/basex/query/QueryParser.java Modified: basex-core/src/main/java/org/basex/query/expr/Catch.java Modified: basex-core/src/test/java/org/basex/query/expr/XQuery4Test.java line 141 -> 151 qe.getMessage() is now replaced with an Empty.VALUE const. Am 11.01.2024 um 12:39 schrieb Martin Honnen: Am 11.01.2024 um 12:37 schrieb Omar Siam : I get: Unknown function: fn:stack-trace() when I use stack-trace(). Ah, sorry, it looks like it is not yet implemented. Wait for Christian to give you hopefully some better info than my ill guess. -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
Re: [basex-talk] Processing stack traces in XQuery code
I get: Unknown function: fn:stack-trace() when I use stack-trace(). I am not sure if this is meant as the equivalent of the stack trace where the error occured. This is what I got in the next catch block which might be anywhere, in my case in an unrelated module that implements a catch all and report functionality. Am 11.01.2024 um 12:19 schrieb Martin Honnen: Am 11.01.2024 um 12:15 schrieb Omar Siam : Hi, Until 10.6 one could find the stack trace of an error in $err:additional. This feature is now gone from BaseX and the documentation. I made my life much easier with reporting stack traces on errors in RestXQ endpoints. Now I don't see how to get that information anymore. Any advice how to get this functionality back? It might be that https://qt4cg.org/specifications/xpath-functions-40/Overview.html#func-stack-trace is already supported (but I haven´t checked). -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
[basex-talk] Processing stack traces in XQuery code
Hi, Until 10.6 one could find the stack trace of an error in $err:additional. This feature is now gone from BaseX and the documentation. I made my life much easier with reporting stack traces on errors in RestXQ endpoints. Now I don't see how to get that information anymore. Any advice how to get this functionality back? Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
Re: [basex-talk] Docker
Hi, If you are a bit adventerous and would like do collaborate with me I can suggest my solution (https://github.com/simar0at/heroku-buildpack-basex) based of heroku buildpacks (the nodejs variety). I use them regularly but I may not always have the build process rock stable. They can be deployed either using Gitlab AutoDevOps (legacy herokuish process, maintained for a little while longer afaik) or you can copy my "minimal port" of that process to GitHub Actions: https://github.com/acdh-oeaw/gl-autodevops-minimal-port. An example of an App deployed to Kubernetes is here: https://github.com/acdh-oeaw/vicav-app Am 05.09.2023 um 14:44 schrieb Diego Alcorta: I see that docker images have not been updated in over 2 years. I was wondering what the best way to deploy basex in kubernetes is. -- Diego Alcorta dalco...@gmail.com - "If patience is a virtue, and ignorance is bliss, you can have a pretty good life if you're ignorant and willing to wait" - Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
Re: [basex-talk] collection('i_do_not_exist')/users/user[@name=()] is () even if i_do_not_exist does not exit
Well it is definitly an optimization that is new to BaseX 10. But I
think I found a way to default to "" where I just passed () from a
variable before.
Am 23.03.2023 um 15:00 schrieb Christian Grün:
Hi again, Omar,
The behavior may be surprising, but it's valid: If it can be detected
at compile time that an expression will never yield results, it can be
simplified and rewritten to an empty sequence, no matter if a
referenced database may possibly exist at runtime.
Hope this helps,
Christian
Omar Siam schrieb am Do., 23. März 2023, 14:49:
Hi,
Either I found an interesting new XQuery feature or a bug:
collection('i_do_not_exist')/some/path[@attr=()] retruns the empty
sequence whereas
collection('i_do_not_exist')/some/path throws an error
err:FODC0002 so does
collection('i_do_not_exist')/ome/path[@attr="i_am_an_attr"]
Am I missing something?
Best regards,
--
Mag. Ing. Omar Siam
Austrian Center for Digital Humanities and Cultural Heritage
Österreichische Akademie der Wissenschaften | Austrian Academy of
Sciences
Stellvertretende Behindertenvertrauensperson | Deputy
representative for disabled persons
Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria
T: +43 1 51581-7295
omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
<http://www.oeaw.ac.at/acdh>
--
Mag. Ing. Omar Siam
Austrian Center for Digital Humanities and Cultural Heritage
Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences
Stellvertretende Behindertenvertrauensperson | Deputy representative for
disabled persons
Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria
T: +43 1 51581-7295
omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
[basex-talk] collection('i_do_not_exist')/users/user[@name=()] is () even if i_do_not_exist does not exit
Hi,
Either I found an interesting new XQuery feature or a bug:
collection('i_do_not_exist')/some/path[@attr=()] retruns the empty
sequence whereas
collection('i_do_not_exist')/some/path throws an error err:FODC0002 so
does collection('i_do_not_exist')/some/path[@attr="i_am_an_attr"]
Am I missing something?
Best regards,
--
Mag. Ing. Omar Siam
Austrian Center for Digital Humanities and Cultural Heritage
Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences
Stellvertretende Behindertenvertrauensperson | Deputy representative for
disabled persons
Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria
T: +43 1 51581-7295
omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
Re: [basex-talk] BaseX 10.5: Patch Release
Hi, Thanks for the new release. Just wanted to try something out with 10.4 but then I can use 10.5 instead :-) https://files.basex.org/releases/10.5/BaseX105.zip returns 403 though at the moment. Am 23.03.2023 um 13:42 schrieb Christian Grün: Hi all, A new patch release of BaseX is available: https://basex.org It gives you a bunch of new XQuery 4 features, advanced performance tweaks and minor bug fixes. As usual, you can have a look at our GitHub commits for more details: https://github.com/BaseXdb/basex/commits/main Have fun, all the best, Christian Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
Re: [basex-talk] xqDoc - potential updates
Hi, You can also have a look at this: https://github.com/acdh-oeaw/openapi4restxq Am 19.10.2022 um 16:17 schrieb Loren Cahlander: Hello folks, I am looking at maybe doing some updates to the xqDoc spec and the codebase. I have code that transforms the xqDoc resources with RestXQ into an OpenAPI 3 API documentation. The things that I want to figure out is: * Figure out how to specify the schema of the parameters and the return * Document the various HTTP return codes and the response content for each return code I would like to get the input from everyone using xqDoc Thank you, Loren Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
Re: [basex-talk] Java 8 and XML Catalog
Hi, Am 07.07.2022 um 17:52 schrieb Majewski, Steven Dennis (sdm7g): BTW: Just to note that Saxon 11.3 has been working fine for me under Basex 9.x, But it requires including the included xmlresolver files in the class path: export CLASSPATH=/usr/local/Cellar/saxon/11.3/libexec/saxon-he-11.3.jar:/usr/local/Cellar/saxon/11.3/libexec/lib/*.jar Oh. I stumbled upon the same thing. But I don't add a directory to the class path where I store a stand alone copy of an XML resolver (and Saxon). There is a jar that seems to contain everything needed and I put it in lib/custom together with saxon-he-11.3.jar (and that is added to the classpath by the startup scripts): lib/custom/Saxon-HE-11.3.jar lib/custom/xmlresolver-4.3.0.jar But I haven’t done any special testing using the resolver from either Saxon or BaseX — Just running my current development code without noticing any problems. Didn't do a formal test either, works for me as described. [...] — Steve M. Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
Re: [basex-talk] RESTXQ: Serve images that are not within the web app dir
Hi, in [1] you can find some pretty generic code that I use (with search and replace for RestXQ paths) in many of my projects to serve anything in some directory related to website served using BaseX. Another example of that code would be [2]. Best regards Omar [1] https://github.com/acdh-oeaw/openapi4restxq/blob/master_basex/swagger-ui.xqm#L18-L120 [2] https://github.com/acdh-oeaw/vicav-app/blob/master/http.xqm#L9-L158 -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
[basex-talk] Something is off with db:enforceindex in 9.7.2.
(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
If I don't use (# db:enforceindex #) {} it works as expected
Best regards
--
Mag. Ing. Omar Siam
Austrian Center for Digital Humanities and Cultural Heritage
Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences
Stellvertretende Behindertenvertrauensperson | Deputy representative for
disabled persons
Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria
T: +43 1 51581-7295
omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
Re: [basex-talk] Adding another document from XQuery evaluation
Hi Eliot, Am 17.05.2022 um 16:34 schrieb Eliot Kimber: This needs to be done in an updating function or module as a separate transaction. If you’re looking for general code on managing sequences of transactions using BaseX’s jobs feature, I created this github project: https://github.com/ekimbernow/basex-orchestration This is a distant relative of my utils module https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/util.xqm . Would you be interested in developing such functions together? Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
Re: [basex-talk] Feature Preview: Caching Module
A very interesting feature for me. I have to admit after I posted my last explanation I found out I already heavily cached data for some search requests in my TEI/XML snippet API. But I hit a low of almost constant 2s for retrieving data using that cache. I probably will do a writeup of that part and try the new caching module. Maybe that is faster. Is there a cache for XQuery code? I work with small snippets and can most of the time choose if something is a literal or passed as an input variable. Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
[basex-talk] Executing optimized XQuery in RestXQ without having to deal with global lock situations
://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/dicts.xqm#L49 [18] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/io/random/DataAccess.java#L184 and other read methods there [19] https://blogs.oracle.com/javamagazine/post/java-nio-nio2-buffers-channels-async-future-callback [20] https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html The view of a file provided by an instance of this class is guaranteed to be consistent with other views of the same file provided by other instances in the same program. [21] https://www.mathematik.uni-marburg.de/~alexmaurer/files/NioVsIo.pdf as an example. May be more recent evaluations of Java 11 or 17 nio or nio.2 performance is better? [22] https://github.com/BaseXdb/basex/blob/master/basex-core/src/test/java/org/basex/io/random/DataAccessTest.java [23] https://github.com/neo4j/neo4j/blob/4.4/community/native/src/main/java/org/neo4j/internal/nativeimpl/LinuxNativeAccess.java Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
Re: [basex-talk] Using error and catch for error paths in REST API endpoint code
Hi Andy,
Hi Christian,
I am happy to have sparked an interesting discussion about modules.
Am 20.04.2022 um 18:47 schrieb Andy Bunce:
Hi Omar,
Thanks for this.
I also enjoyed your earlier message re:
>I tried to get creative at finding a way to optimize querying the data without having many long lasting
global lock situations
I have been in a similar situation and discovered how alternative code
approaches using different builtin functions can have very different
lock profiles.
I am hopeful that #[2063] will greatly reduce locking issues.
I will try my "worst case" Data with my API with new BaseX release. I
will post any changes I see.
>Is there any sane way to get a QName with an unknown prefix of an
error as a string like in the catch all handler and resolve it against
all prefix-URI mappings known in some XQuery program?
I will try to integrate your idea, Andy, into the generic error handler.
This routine just gets the string representation of the error with
prefix:value and not a Clark notation like {urn:_}value
I hope this is not a big performance hit.
If you feel like it a PR would be much appreciated.
I will post more thoughts on my modules soon, just have to prepare some
workshop right now.
Best regards
--
Mag. Ing. Omar Siam
Austrian Center for Digital Humanities and Cultural Heritage
Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences
Stellvertretende Behindertenvertrauensperson | Deputy representative for
disabled persons
Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria
T: +43 1 51581-7295
omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
[basex-talk] Using error and catch for error paths in REST API endpoint code
standards. [17] By the way: developing this started on BaseX but then we had similar needs in another open-source XML database existing today so I tried to port the code. Not only worked that rather well (probably also because I know the XQuery needed there rather well too), I also had a few new ideas and added them back to the BaseX version. May be there is some left over code in the module at the moment that reimplements some helpful, non-standard XQuery functions for this reason. The code is somewhat portable. A few thoughts: * Some say that it is necessary for security purposes to disable any stack traces in production environments. I am not really believing this does much good. But if one does not want to hard code a Boolean switch in the modules source code: What would be the fastest external source one could use in terms of compile, optimizing and execution time? Are stack traces not available as “$err:additional” when RESTXQERRORS are switched off? * Is there any sane way to get a QName with an unknown prefix of an error as a string like in the catch all handler and resolve it against all prefix-URI mappings known in some XQuery program? [1] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/users.xqm#L90-L152 [2] https://github.com/acdh-oeaw/vicav-app/blob/master/http.xqm#L24-L55 [3] https://api-tools.getlaminas.org/ [4] https://api-tools.getlaminas.org/documentation/modules/api-tools-api-problem [5] https://datatracker.ietf.org/doc/html/rfc7807 [6] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/api-problem/api-problem.xqm#L3 [7] https://docs.basex.org/wiki/XQuery_3.0#Try.2FCatch [8] https://docs.basex.org/wiki/RESTXQ#Catch_XQuery_Errors [9] https://docs.basex.org/wiki/JSON_Module#Direct [10] https://docs.basex.org/wiki/Request_Module#request:header [11] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/tests/api-problem-rest-test.xqm#L21 [12] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/tests/http.xqm#L44 [13] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/tests/http.xqm#L100-L105 [14] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/dicts.xqm#L47-L51 [15] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/api-problem/api-problem.xqm#L179-L188 [16] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/tests/api-problem-rest-test.xqm#L20 [17] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/api-problem/api-problem.xqm#L487 Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
[basex-talk] An automated workflow for creating tested and sustainable REST API/RestXQ containers that use BaseX
really show you the gitlab CI/CD workflow I made the buildpack for in action, you have to take my word for it or try it yourself on some gitlab instance. Reenabling the old herokuish workflow for build (AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED=false) and referencing my BaseX buildpack (BUILDPACK_URL=https://github.com/simar0at/heroku-buildpack-basex) will integrate BaseX into the AutoDevOps workflow. But you can also see it running in some of our public projects on github.com. There just a pretty stereotypical .github/workflows YAML file is needed and you can use the github.com Actions just fine [15]. That also has the advantage that one can utilize the infrastructure on github.com instead of providing some compute resources. The package might be still a little rough around the edges but I use it now for a few projects of mine (and my institution). I hope someone finds this useful. There are probably a lot of things that need to be fine tuned and I would be happy to work together to get this package better with time. [1] https://docs.gitlab.com/ee/topics/autodevops/ [2] https://docs.basex.org/wiki/Unit_Module [3] https://docs.basex.org/wiki/Unit_Module#Example [4] for one of the solutions existing for Node.js check out https://www.cypress.io/ . That is what I used recently. [5] https://github.com/gliderlabs/herokuish [6] https://devcenter.heroku.com/articles/buildpacks [7] https://buildpacks.io/ [8] https://github.com/simar0at/heroku-buildpack-basex [9] https://github.com/acdh-oeaw/zuludict-app/blob/master/package.json#L6-L12 [10] https://github.com/acdh-oeaw/vleserver_basex/blob/main/package.json#L19-L25 [11] https://github.com/acdh-oeaw/zuludict-app/tree/master/deployment [12] https://github.com/acdh-oeaw/zuludict-app/blob/master/deployment/deploy-zuludict-data.bxs [13] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/package.json#L6-L8 [14] https://github.com/acdh-oeaw/vicav-app/tree/master/cypress/integration [15] https://github.com/acdh-oeaw/vleserver_basex/blob/main/.github/workflows/build-vleserver_basex.yml Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
[basex-talk] I would like to share some code I use for easier RestXQ development
Hi list, I am using BaseX here at my institution a lot as an XML database that provides data using RESTful APIs [1] or serves HTML, JS and CSS with RestXQ which in turn uses HTTP requests to fetch more data [2]. Our main use of BaseX databases is about TEI/XML encoded dictionaries of various sizes. That means we mainly want to search for, get and change small tei:entry parts in larger TEI/XML documents. BaseXwas amazingly stable (compared to some other open source XML database existing today) over the last few years and has a very solid set of built in functions that almost always are sufficient to get a job done. I also like BaseX because I am pretty sure I always could understand why things did not work and what I can do about it even without doing any Java programming (although I saw some weirdness or the other over the last few years). I created or ported some XQuery modules [3][4][6][7] and a containerization environment [5] that make my life easier and I would like to share them here and maybe have a discussion about my implementations and if others can make use of them and how (for example can these [6][7] be expath packages, I see some obstacles in the way RestXQ annotations work). I will try to give an introduction to each of the modules in some separate mails. Finally, I now write to the list as I have a performance problem with a CRUD API I created [1] for the task mentioned above when using it with a dataset that is about 7GB looking at the BaseX databases that make it up. This API uses many of the modules and techniques I came up with so I thought it might be helpful to first talk about those parts. I hope that they may be useful to others as well. I tried to get creative at finding a way to optimize querying the data without having many long lasting global lock situations and having BaseX using indexes as much as possible (this started before the db:enforceindex pragma was introduced and still works for me as expected without it) while still writing the RESTful API in BaseX' implementation of RestXQ. That is why I created [7]. It heavily uses (abuses?) BaseX' jobs module. It allows me to query in smaller BaseX databases in parallel and present them as if they were one big XML DB, which vastly improves performance on update and reindex, to a point. Still there is a file based lock (or is it even class based?) [8], I think the JVM profiling tells me, that severely limits the number of (read) operations that can be done over the API without the user having to wait so long they think the operation failed. This is a multi threading problem as I see it. Or maybe I overlooked something that would solve my problems without all the creative stuff I tried? That probably will be obvious when I have explained the current implentation in more detail which I intend to do in the next few days. [1] https://vle-curation.acdh.oeaw.ac.at/openapi/, https://github.com/acdh-oeaw/vleserver_basex [2] https://vicav.acdh.oeaw.ac.at/, https://github.com/acdh-oeaw/vicav-app [3] https://github.com/acdh-oeaw/openapi4restxq [4] https://github.com/acdh-oeaw/api-problem4restxq [5] https://github.com/simar0at/heroku-buildpack-basex [6] https://github.com/acdh-oeaw/vicav-app/blob/master/http.xqm, https://github.com/acdh-oeaw/openapi4restxq/blob/master_basex/swagger-ui.xqm [7] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/util.xqm [8] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/io/random/DataAccess.java#L184 and other read mehods there Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at |www.oeaw.ac.at/acdh
Re: [basex-talk] docker, creating and deploying the dba docker-image
Hi Rob, What do you actually want to achieve here? If you want to compile xquery files to protect your IP I have no idea how to do that. Saxon has such a feature in a limited way if you buy PE or EE if I remeber correctly. You can call Saxon using xslt:transform. I do a lot of docker stuff and I * used a JVM image and mounted everything BaseX (so the zip distribution including any xquery code I added) * tried to create a buildpack that downloads everything needed and creates a container form that: https://github.com/simar0at/heroku-buildpack-basex This is a https://github.com/gliderlabs/herokuish buildpack (usabel with gitlab CI/CD). It is still rough around the edges Maybe you want something in between? You probably will need to mount /data (which also contains the logs), and /webapp containing any code for the (rest based?) services you want to provide. Best regards Omar
Re: [basex-talk] EMail module and javax.activation.DataSource
Hello, Are you using JDK 11 (or newer)? javax.activation is no longer part of the JDK but the jar is available separately. I stumbled upon this in exist-db but maybe it is the same problem here? So JDK 1.8 should work and adding the jar https://search.maven.org/remotecontent?filepath=javax/activation/activation/1.1.1/activation-1.1.1.jar to the lib or lib/custom folder should work to. Best regards Omar Siam Am 25.08.2021 um 13:57 schrieb dudz...@yahoo.com: Dear all, I am trying to use a basex Email module http://files.basex.org/modules/org/basex/modules/email/EMail.jar. I downloaded a new basex instance (9.6 version) and added this jar to lib directory. Based on the xquery/email.xq file I created the following file: import module namespace email = 'http://basex.org/modules/email/EMail'; let $mail := myuser@machine-office myuser@machine-office localhost Hello! Enjoy return email:send($mail) While running the file I obtain: Stopped at /home/myuser/Downloads/basex/file, 12/13: [XPTY0004] javax.activation.DataSource. Caused by: org.basex.modules.email.EMail:send(element()). What's the problem? Shouldn't I use this module? BR -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
Re: [basex-talk] Using other Java runtimes?
Hi, I switched to https://www.azul.com/downloads/zulu-community/ for Windows and Mac shortly after Oracle announced their license change. On Linux I use the built in JDK. Tried Zulu Community on Linux as container for some legacy Java that had to be run or built with some of the older versions 8 and before. Never had any problems. It seems BaseX is a software that doesn't use anything only Oracle has in their distribution. I start to forget about Oracle as primary source of Java. Any OpenJDK is good these days I think. I hope that is what they intended to achieve. Best regards Omar Am 16.04.2021 um 23:39 schrieb Peter Villadsen: > > Guys, > > > > I was wondering what the options are for running BaseX on top of a > runtime that is different from the Oracle one? I see Oracle is > aggressively monetizing the Java runtime, but I know there are > alternatives out there. Have any been tested with BaseX? > > > > > > Best Regards > > > > Peter Villadsen > > Principal Architect > > Microsoft Business Applications Group > > >
Re: [basex-talk] Preferences do not open, Exception in thread AWT-EventQueue-0
Hi, I like Debian very much. The package management is one of the best and most to all of the packages I use are well enough maintained and work and update as expected when using the buster-backports repository. It pains me to read, not for the first time, that the Debian BaseX package seems to be on a kind of less maintained base than the files downloadable at basex.org and no one cares about updates or better backports. Pure unmodified Debian stable has a tendency of aging. But may be some people develop their code exclusivly for Debian 10 stable packages. That is a valid approach. I have no clue if there is any list of packages activly maintained during the Debian stable lifecycle. I guess not. Ubuntu has such a list and a tool to check the setup for the maintenance state for each package. The ubuntu list of mnaintained packages is surprisingly short if you look at the over all huge number of packages available. For me this is less about the well maintained vanilla distribution of BaseX, which I use with any JDK on any OS, including Debian 10, then with the BaseX Debian package 9.0.1+ds-1 that is not part of the regular release process of basex.org and so outdated and furthermore created in a manner that is especially hard to understand and maintain seperately from the rest of Debian. Maybe I am wrong, but is the current maintainer of the Debian packages actually on this list? It seems to me we get error reports here which are totally lost on the person who might do something about the problems. Best regards Omar Siam Am 17.03.2021 um 19:06 schrieb Joris Lambrecht: Thanks for the perspective. Here's what i hope comes across as a polite rant as a reply. It aches me and i notice it aches others everywhere. I spent 20 years on and off with Linux, starting with a CT magazine CD somewhere late 90's. I've really grown stone cold tired of the package management nonsense with unmaintained, poorly maintained software. Except for the big packges like desktops that is for which no sane compilation procudure exists. Also get irritated with the so called distro specific narrative. There are exactly 2 systems, Debian and RedHat of which Debian is universally compatible and Redhat is not so much. It should not be an issue really. This implies developing compatible with the RedHat system is alwasy going to work on Debian, not vice versa. When it comes to package managers almost none of the work well or if they work well there is no sensible package maintenance happening. Or they build a great set of software like Sabayon Linux and stop the disro. The times i found myself downloading source and compiling from scratch to have all features and flags enabled became too many. Yet every time i go back almost to not using package managers. Such as I end up with now with BaseX By now i'm contemplating a no-package manager custom Linux distroy with update scripts for my personal choice of tools. Simply to avoid the cruft and 99% meaningless software in most distributions. Maybe i build it all on BaseX to manage configuration, compilation and other stuff. I'm a niche user who also happens to be picky and mostly atypical as an IT profile. The future will probably bring hard choices and hard work for me because of it. Br, Joris
[basex-talk] Simple xqdoc to HTML converter
Hi, I brushed up the 2006 xqdoc to HTML XQuery a bit to document some of my code. Maybe this is also interesting to others. https://github.com/acdh-oeaw/xqdoc I also found https://github.com/lcahlander/xqDoc-eXist-db mentioned on this mailing list. I plan to look into getting this to work on BaseX. Best regards Omar Siam
Re: [basex-talk] where is the "lib" folder?
Hi, Am 23.11.2020 um 10:57 schrieb Nicholas: Is there even a lib directory? https://unix.stackexchange.com/a/621089/101935 My understanding is that, for example, a JDBC driver can be "added" to BaseX itself. Perhaps this is incorrect. Adding JAR libraries is a matter of setting the CLASSPATH correctly. It seems the Debian package is very minmalistic. I have to admit I never used it. The lib folder and tho lib/custom folder are obvious if you download the ZIP distribution (see https://basex.org/download/). If you only work with the (GUI) single jar file or as it seems the maven distribution here, things are more complicated if they work at all. You can look at the start scripts (probably shell scripts) and compare them to what the ZIPed versions do. Perhaps you can try the ZIP version first before searching any further. Best regards Omar Siam
Re: [basex-talk] Paging results of an XQuery-Search
Perhaps you want to use fn:subsequence($hits, $start, $num)?
But of course if you have millions of hits that is not efficient enough.
Am 25.09.2020 um 13:23 schrieb Günter Dunz-Wolff:
Hi all,
is it possible to present the results of an XQuery search in parts of 50? In
order to speed up the search, only the first 50 results should be displayed and
only if the user is interested in further results, 50 new results each should
be displayed. Is this even possible? I did not find anything about it in the
documentation.
Example:
let $collection := collection("data")
let $hits := $collection//*:s[.//text() contains text {$query_string}]
let $count_hits := count($hits)
return
{
if $count_hits <= 50 then
{for $hit in $hits
return
{$hit}
else
???
}
Thanks for any advice.
Best regards
Guenter
--
Mag. Ing. Omar Siam
Austrian Center for Digital Humanities and Cultural Heritage
Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences
Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria
T: +43 1 51581-7295
omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
Re: [basex-talk] Enabling Xerces2 XML Schema processor
You need to download the Xerces2 Java 2.12.1 (XML Schema 1.1) variant: Xerces-J-bin.2.12.1-xml-schema-1.1.zip . Am 09.07.2020 um 17:25 schrieb Yasir B: Hello, Very new to this - seeking help on how to enable the Xerces2 XML Schema processor as described in https://docs.basex.org/wiki/Validation_Module#XML_Schema_Validation I downloaded the Xerces2 Java 2.12.1, but could not find the following jar files: org.eclipse.wst.xml.xpath2.processor_1.2.0.jar, cupv10k-runtime.jar Thank you -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.s...@oeaw.ac.at | www.oeaw.ac.at/acdh
Re: [basex-talk] Random Numbers from Random.org
In the XQuery data model doc() do not change while a query is evaluated.
So a doc() read once can in will be cached for that query. You may get
what you want if you use %non-deterministic.
The built in random functions work like that.
Am 03.06.2020 um 18:26 schrieb Leo Studer:
Hello
I try to use random numbers from random.org.
Here my xpath
let $urand :=
function(){doc("https://www.random.org/integers/?num=1min=0max=10col=1base=10format=htmlrnd=new;)//*:pre/number()
div 10}
return
for $i in 1 to 10 return
$urand()
Which brings 10 time the same value, however, every run time a different one.
What is wrong with this code?
Thanks in advance for your valuable input
Yours
Leo
Re: [basex-talk] I am confused how to specify multiple possible reponses in RestXQ with %rest:produces
That is much better now.
There are two further improvement: I think %rest:consumes and
%rest:produces can be specified pretty much alike so for both the
statement "Multiple types can either be specified by a single or by
multiple annotations." is true. Unfortunately you only say that in the
subparagraph about Pruducing Data. If I only care about Consuming Data
right now I probably will not read this.
Besides the example for consuming data really needs a function parameter
else I would start wondering by what magic the content I send in the
body is represented in the function.
Best regards
Omar Siam
Am 11.05.2020 um 18:31 schrieb Christian Grün:
Hi Omar,
Your question was if multiple mime types need to be specified via a
single or multiple annotations, right?
I have added some examples to the Content Negotiation section; I hope
it’s more comprehensible now.
Hope this helps,
Christian
On Mon, May 11, 2020 at 3:54 PM Omar Siam wrote:
Hi,
Today someone on the exist-open mailing list pointed out how they
interpret %rest:produces: If your function produces more types of
responses you specify all mime types just the same as with
%rest:consumes: %rest:poroduces("application/xml", "application/json",
...) That is as a list of possible reponses all in one annotation.
The BaseX docs at https://docs.basex.org/wiki/RESTXQ#Content_Negotiation
however are ambigous about this I think. There is an example there with
just two %rest:produces annotations on two lines with only one mime type
(and a qs value). I think that lead me to belief that to specify
multiple possible responses of different mime types you repeat
%rest:produces.
Rereading the section these two example annotations could also ilustrate
the use of %rest:produces in two different functions.
Can you please clarify this if possible?
Best regards
Omar Siam
[basex-talk] I am confused how to specify multiple possible reponses in RestXQ with %rest:produces
Hi,
Today someone on the exist-open mailing list pointed out how they
interpret %rest:produces: If your function produces more types of
responses you specify all mime types just the same as with
%rest:consumes: %rest:poroduces("application/xml", "application/json",
...) That is as a list of possible reponses all in one annotation.
The BaseX docs at https://docs.basex.org/wiki/RESTXQ#Content_Negotiation
however are ambigous about this I think. There is an example there with
just two %rest:produces annotations on two lines with only one mime type
(and a qs value). I think that lead me to belief that to specify
multiple possible responses of different mime types you repeat
%rest:produces.
Rereading the section these two example annotations could also ilustrate
the use of %rest:produces in two different functions.
Can you please clarify this if possible?
Best regards
Omar Siam
Re: [basex-talk] Storing xml message
If you pass some XML in to your query as text (because that is the only
thing you can have inside an attribute) you have to do a parse-xml (or
parse-xml-fragment if its more like a sequence) and insert that.
Am 21.04.2020 um 12:33 schrieb Ramzi Hammouda:
Hello,
First, thanks for the effort you’re putting it to help BaseX users.
I'm Using POST method to update BaseX database, I want to insert an
xml node to an existing document calastone.xml in calastone database ,
if the node contains '<', it returns an error *" Line 6): The value of
attribute "value" associated with an element type "variable" must not
contain the '<' character."*
code:
let $message := 'CTN53'
return insert nodes $message as last into
doc("calastone/calastone.xml")
the same code without '<' adds the text correctly.
how to solve this problem?
Note that this is a question I posted on Stackoverflow, Link
.
Christian Grun suggested using || and | for attributes and
CDATA for text nodes. After applying that solution, the xml stored is
in this form IdCTN53/Id|||
|This is not what I’m looking for. Is there any other more convenient
solutions to store XML messages with Rest APIs? I know PUT method
works fine but it overwrites the content of the document.|
|Thank you.|
P/Respectons ensemble l'environnement. N'imprimez ce message que si
nécessaire. //Let's respect the environment together. Only print this
message if necessary./
Re: [basex-talk] ft search with wildcards does not work with higher Unicode characters
Hi Christian,
I deployed the latest beta and I am quite happy with the performance and
all my problems seem to be fixed. I did not test without index yet.
Maybe I have time to do this later this week with a bigger dataset.
Best regards
Omar Siam
Am 09.02.2020 um 11:20 schrieb Christian Grün:
Hi Omar,
Full-text queries with wildcards should now work with diacritical
strings as well, both with the index and with sequential requests.
Furthermore, queries with more than 1 dot in the wildcard pattern are
now evaluated by the index if one is available. I’m still not sure if
this is a good decision, as query strings with wildcards might lead to
an extensive number of atomic index requests, and might get slower
than full XML scans. If you observe that your wildcard queries are
processed faster if you disable the full-text index, feel free to
report this back to us.
A new snapshot is available. Have fun,
Christian
PS: The xs:string cast bug is fixed as well.
On Thu, Feb 6, 2020 at 3:16 PM Christian Grün wrote:
Great! Confirmed and registered [1].
[1] https://github.com/BaseXdb/basex/issues/1802
On Thu, Feb 6, 2020 at 3:03 PM Omar Siam wrote:
I think I've got it: for reasons I am not 100% sure about right now a usually
select diacritics: true()
So the following does not work as expected:
db:open('testdata')//*[text() contains text 'r.{1,1}ḥ' using wildcards
using diacritics sensitive]
db:open('testdata')//*[text() contains text 'r.ḥ' using wildcards using
diacritics sensitive]
Am 06.02.2020 um 14:40 schrieb Christian Grün:
I can try. But what you tell me and what I saw with the unit test function I
guess the GUI does something that can't be observed when using scripts.
Maybe some options are still active in your GUI that are not reflected
in the script?
Does the script work if you run it on command line?
Unfortunately I get new problems with 9.3.2 Beta:
…most probably related to recent typecheck optimizations that we
recently introduced [1]; sorry for that. If you need a very short-term
alternative, you can use string() instead of xs:string().
[1] https://github.com/BaseXdb/basex/issues/1801
declare function local:maptest() as map(xs:string, xs:string) {
let $xml := <_ test="test"/>
return map{xs:string($xml/@test): xs:string('test')}
};
local:maptest()
stopped working.
[XPTY0004] Cannot convert map(*) to map(xs:string, xs:string): map { "test":
"test" }.
And I get more new type conversion related errors that were not reported
before. This is temporary I hope?
Best regards
Omar
Am 06.02.2020 um 14:11 schrieb Christian Grün:
Looks we are getting closer. I have attached a little command script;
do you think you can adapt it such that it captures the potential bug?
On Thu, Feb 6, 2020 at 1:51 PM Omar Siam wrote:
The following XQuery run in the GUI (pulled from github and built a few minutes
ago from source)
ft:tokens('testdata'),
ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '--'
,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards]
yields
rwḥ
--
with collection('testdata')
<_>
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-tunis-vicav">rwḥ
But the gh1800() test changed like this:
final String text = "999 aa rwḥ";
[...]
query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text);
works.
Am 06.02.2020 um 13:45 schrieb Christian Grün:
I just tried to use the gh1800 test to replicate my problem and it does
not show there. It fails using the GUI.
I need your help: What does not show there? What fails, what happens?
Am 06.02.2020 um 13:35 schrieb Christian Grün:
Hi Omar,
Yes, that seems to solve the problem partly. Using wildcards now yields the
same result as no wildcards.
Glad to hear.
But if there is a complex unicode character in the search string, "." for one
character looses its meaning.
…
Would you like a PR for the test gh1800 using complex unicode characters?
A little test case would be helpful indeed. It seems to be a different issue:
• The first expression is evaluated without the full-text expression.
The reason is that the full-text index algorithms are limited to basic
regular expressions; not all of them can be answered by an index (and
'r{1,1}' is currently not detected as being identical to `r.`). If I
remember correctly, the index will not be accessed either if a pattern
starts with `.*` (this pattern would lead to a full index scan).
• The second expression is rewritten for index access. I tried to
build a little co
Re: [basex-talk] ft search with wildcards does not work with higher Unicode characters
I think I've got it: for reasons I am not 100% sure about right now a
usually select diacritics: true()
So the following does not work as expected:
db:open('testdata')//*[text() contains text 'r.{1,1}ḥ' using wildcards
using diacritics sensitive]
db:open('testdata')//*[text() contains text 'r.ḥ' using wildcards using
diacritics sensitive]
Am 06.02.2020 um 14:40 schrieb Christian Grün:
I can try. But what you tell me and what I saw with the unit test function I
guess the GUI does something that can't be observed when using scripts.
Maybe some options are still active in your GUI that are not reflected
in the script?
Does the script work if you run it on command line?
Unfortunately I get new problems with 9.3.2 Beta:
…most probably related to recent typecheck optimizations that we
recently introduced [1]; sorry for that. If you need a very short-term
alternative, you can use string() instead of xs:string().
[1] https://github.com/BaseXdb/basex/issues/1801
declare function local:maptest() as map(xs:string, xs:string) {
let $xml := <_ test="test"/>
return map{xs:string($xml/@test): xs:string('test')}
};
local:maptest()
stopped working.
[XPTY0004] Cannot convert map(*) to map(xs:string, xs:string): map { "test":
"test" }.
And I get more new type conversion related errors that were not reported
before. This is temporary I hope?
Best regards
Omar
Am 06.02.2020 um 14:11 schrieb Christian Grün:
Looks we are getting closer. I have attached a little command script;
do you think you can adapt it such that it captures the potential bug?
On Thu, Feb 6, 2020 at 1:51 PM Omar Siam wrote:
The following XQuery run in the GUI (pulled from github and built a few minutes
ago from source)
ft:tokens('testdata'),
ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '--'
,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards]
yields
rwḥ
--
with collection('testdata')
<_>
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-tunis-vicav">rwḥ
But the gh1800() test changed like this:
final String text = "999 aa rwḥ";
[...]
query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text);
works.
Am 06.02.2020 um 13:45 schrieb Christian Grün:
I just tried to use the gh1800 test to replicate my problem and it does
not show there. It fails using the GUI.
I need your help: What does not show there? What fails, what happens?
Am 06.02.2020 um 13:35 schrieb Christian Grün:
Hi Omar,
Yes, that seems to solve the problem partly. Using wildcards now yields the
same result as no wildcards.
Glad to hear.
But if there is a complex unicode character in the search string, "." for one
character looses its meaning.
…
Would you like a PR for the test gh1800 using complex unicode characters?
A little test case would be helpful indeed. It seems to be a different issue:
• The first expression is evaluated without the full-text expression.
The reason is that the full-text index algorithms are limited to basic
regular expressions; not all of them can be answered by an index (and
'r{1,1}' is currently not detected as being identical to `r.`). If I
remember correctly, the index will not be accessed either if a pattern
starts with `.*` (this pattern would lead to a full index scan).
• The second expression is rewritten for index access. I tried to
build a little command script (test.bxs), but it doesn’t seem to
reflect the case you encountered:
set ftindex true
create db test rwḥ
xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards]
xquery /*[text() contains text 'r.ḥ' using wildcards]
close
Could you extend this example script a little, such that it
demonstrates what goes wrong?
Thanks in advance,
Christian
Re: [basex-talk] ft search with wildcards does not work with higher Unicode characters
Hi,
I can try. But what you tell me and what I saw with the unit test
function I guess the GUI does something that can't be observed when
using scripts.
Unfortunately I get new problems with 9.3.2 Beta:
declare function local:maptest() as map(xs:string, xs:string) {
let $xml := <_ test="test"/>
return map{xs:string($xml/@test): xs:string('test')}
};
local:maptest()
stopped working.
[XPTY0004] Cannot convert map(*) to map(xs:string, xs:string): map { "test":
"test" }.
And I get more new type conversion related errors that were not reported
before. This is temporary I hope?
Best regards
Omar
Am 06.02.2020 um 14:11 schrieb Christian Grün:
Looks we are getting closer. I have attached a little command script;
do you think you can adapt it such that it captures the potential bug?
On Thu, Feb 6, 2020 at 1:51 PM Omar Siam wrote:
The following XQuery run in the GUI (pulled from github and built a few minutes
ago from source)
ft:tokens('testdata'),
ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '--'
,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards]
yields
rwḥ
--
with collection('testdata')
<_>
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-tunis-vicav">rwḥ
But the gh1800() test changed like this:
final String text = "999 aa rwḥ";
[...]
query("ft:search('" + NAME + "', 'r.ḥ', " + options + ")", text);
works.
Am 06.02.2020 um 13:45 schrieb Christian Grün:
I just tried to use the gh1800 test to replicate my problem and it does
not show there. It fails using the GUI.
I need your help: What does not show there? What fails, what happens?
Am 06.02.2020 um 13:35 schrieb Christian Grün:
Hi Omar,
Yes, that seems to solve the problem partly. Using wildcards now yields the
same result as no wildcards.
Glad to hear.
But if there is a complex unicode character in the search string, "." for one
character looses its meaning.
…
Would you like a PR for the test gh1800 using complex unicode characters?
A little test case would be helpful indeed. It seems to be a different issue:
• The first expression is evaluated without the full-text expression.
The reason is that the full-text index algorithms are limited to basic
regular expressions; not all of them can be answered by an index (and
'r{1,1}' is currently not detected as being identical to `r.`). If I
remember correctly, the index will not be accessed either if a pattern
starts with `.*` (this pattern would lead to a full index scan).
• The second expression is rewritten for index access. I tried to
build a little command script (test.bxs), but it doesn’t seem to
reflect the case you encountered:
set ftindex true
create db test rwḥ
xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards]
xquery /*[text() contains text 'r.ḥ' using wildcards]
close
Could you extend this example script a little, such that it
demonstrates what goes wrong?
Thanks in advance,
Christian
Re: [basex-talk] ft search with wildcards does not work with higher Unicode characters
The following XQuery run in the GUI (pulled from github and built a few
minutes ago from source)
ft:tokens('testdata'),
ft:search('testdata', 'r.ḥ', map {'wildcards': true()})/.., '--'
,collection('testdata')//*[text() contains text 'r.ḥ' using wildcards]
yields
rwḥ
--
with collection('testdata')
<_>
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-tunis-vicav">rwḥ
But the gh1800() test changed like this:
final String text ="999 aa rwḥ";
[...]
query("ft:search('" +NAME +"', 'r.ḥ', " + options +")", text);
works.
Am 06.02.2020 um 13:45 schrieb Christian Grün:
I just tried to use the gh1800 test to replicate my problem and it does
not show there. It fails using the GUI.
I need your help: What does not show there? What fails, what happens?
Am 06.02.2020 um 13:35 schrieb Christian Grün:
Hi Omar,
Yes, that seems to solve the problem partly. Using wildcards now yields the
same result as no wildcards.
Glad to hear.
But if there is a complex unicode character in the search string, "." for one
character looses its meaning.
…
Would you like a PR for the test gh1800 using complex unicode characters?
A little test case would be helpful indeed. It seems to be a different issue:
• The first expression is evaluated without the full-text expression.
The reason is that the full-text index algorithms are limited to basic
regular expressions; not all of them can be answered by an index (and
'r{1,1}' is currently not detected as being identical to `r.`). If I
remember correctly, the index will not be accessed either if a pattern
starts with `.*` (this pattern would lead to a full index scan).
• The second expression is rewritten for index access. I tried to
build a little command script (test.bxs), but it doesn’t seem to
reflect the case you encountered:
set ftindex true
create db test rwḥ
xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards]
xquery /*[text() contains text 'r.ḥ' using wildcards]
close
Could you extend this example script a little, such that it
demonstrates what goes wrong?
Thanks in advance,
Christian
Re: [basex-talk] ft search with wildcards does not work with higher Unicode characters
I just tried to use the gh1800 test to replicate my problem and it does
not show there. It fails using the GUI.
Am 06.02.2020 um 13:35 schrieb Christian Grün:
Hi Omar,
Yes, that seems to solve the problem partly. Using wildcards now yields the
same result as no wildcards.
Glad to hear.
But if there is a complex unicode character in the search string, "." for one
character looses its meaning.
…
Would you like a PR for the test gh1800 using complex unicode characters?
A little test case would be helpful indeed. It seems to be a different issue:
• The first expression is evaluated without the full-text expression.
The reason is that the full-text index algorithms are limited to basic
regular expressions; not all of them can be answered by an index (and
'r{1,1}' is currently not detected as being identical to `r.`). If I
remember correctly, the index will not be accessed either if a pattern
starts with `.*` (this pattern would lead to a full index scan).
• The second expression is rewritten for index access. I tried to
build a little command script (test.bxs), but it doesn’t seem to
reflect the case you encountered:
set ftindex true
create db test rwḥ
xquery /*[text() contains text 'r.{1,1}ḥ' using wildcards]
xquery /*[text() contains text 'r.ḥ' using wildcards]
close
Could you extend this example script a little, such that it
demonstrates what goes wrong?
Thanks in advance,
Christian
Re: [basex-talk] ft search with wildcards does not work with higher Unicode characters
Hi,
Yes, that seems to solve the problem partly. Using wildcards now yields
the same result as no wildcards.
But if there is a complex unicode character in the search string, "."
for one character looses its meaning.
collection('testdata')//*[text() contains text 'r.{1,1}ḥ' using wildcards]
works but
collection('testdata')//*[text() contains text 'r.ḥ' using wildcards]
does not. testdata is just my result from below.
Would you like a PR for the test gh1800 using complex unicode characters?
The example in the spec
//book[@number="1"]/p[text() contains text "w.ll" using wildcards]
works using this XML:
will turn
last will
will find
well done
Best regards
Omar
Am 05.02.2020 um 19:59 schrieb Christian Grün:
Dear Omar,
At about the same time when you wrote this, we have fixed a little bug
that occurred with the wildcards option [1]. Could you have a look at
the latest snapshot [2] and report back to us if it resolves the issue?
Thanks in advance,
Christian
[1] https://github.com/BaseXdb/basex/issues/1800
[2] http://files.basex.org/releases/latest/
Omar Siam mailto:omar.s...@oeaw.ac.at>> schrieb
am Mi., 5. Feb. 2020, 17:02:
Hi,
I just came across this strange behavior
collection('dc_tunico')//*[text() contains text 'rwḥ' using wildcards]
yields nothing vs
collection('dc_tunico')//*[text() contains text 'rwḥ']
yields the correct result
http://www.tei-c.org/ns/1.0; <http://www.tei-c.org/ns/1.0> type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; <http://www.tei-c.org/ns/1.0> type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; <http://www.tei-c.org/ns/1.0> type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; <http://www.tei-c.org/ns/1.0> type="root"
xml:lang="ar-aeb-x-tunis-vicav">rwḥ
Any ideas why this is the case?
Best regards
Omar
[basex-talk] ft search with wildcards does not work with higher Unicode characters
Hi,
I just came across this strange behavior
collection('dc_tunico')//*[text() contains text 'rwḥ' using wildcards]
yields nothing vs
collection('dc_tunico')//*[text() contains text 'rwḥ']
yields the correct result
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-vicav">rwḥ
http://www.tei-c.org/ns/1.0; type="root"
xml:lang="ar-aeb-x-tunis-vicav">rwḥ
Any ideas why this is the case?
Best regards
Omar
Re: [basex-talk] Load LibreOffice- and Word-documents?
Hi Ben This will be problematic with real world docx files at least. The text in there can be split into numerous tags disregarding any word boundaries depending on the edit history of the document. As BaseX has no means to ignore inline elements in the index this will always be a rather slow process. To formulate an XQuery will be a complicated task. Unless you clean up the docx XML beforehand that is. Omar Am 28.01.2020 um 14:01 schrieb Ben Engbers: Hi, While we were discussing possible usecases for basex, a colleague asked me if it is also possible to load libreoffice and Word documents into Basex and then perform full-text analysis on them. In essence, these are both XML files, so it should be possible. Does anybody have experience with this? Ben
[basex-talk] HTTPS requests using the HTTP Client Module with self signed (invlaid) certificates
Hi, I just searched for a way to turn off certificate verification for the HTTP Client xquery module. Turns out I have to use Client/Server Architecture option IGNORECERT. Although that is somewhat logical I miss a clear statement in the docs. Thanks for the great software! Omar Siam
Re: [basex-talk] slow search performance across multiple databases
Hi,
I have trouble seeing the original second query.
But if I got it correct then the problem is that to have BaseX
automatically rewrite for indexing you have to supply the DB you want to
search in as string literal. That is db:open("DB") and not let $db :=
"DB" db:open($db). Or for with multiple dbs for that matter.
Maybe I am wrong and the enforceindex pragma solves that now. I have to
admit my code predates the pragma and always uses string literals when
accessing a database.
Best regards
Omar
Am 14.01.2020 um 13:33 schrieb Braunschweig, Björn:
Dear Christian,
Thank you for your quick response.
Could you tell me why the following two queries have a difference in behaviour
so that the first one apparently uses the text index [1] and the second one [2]
does not?
Your second suggestion with db:text is nice but I cannot figure out how to
efficiently extract other informations like something from the root element
without crawling the whole database and performing a full table scan so to say.
Regards,
Björn
[1]
declare namespace lmerFile = "http://www.ddb.de/LMERfile;;
count(
(# db:enforceindex #) {
let $dbs := ("00")
for $db in $dbs
for $doc in db:open($db)
where $doc//lmerFile:format/text() =
'urn:diasid:fty:kopal:00'
return $doc
})
Compiling:
- merge steps: descendant::element(lmerFile:format)
- rewrite for to let: for $db_1 in $dbs_0
- inline $dbs_0
- inline $db_1
- pre-evaluate db:open(database[,path]) to document-node() sequence: db:open("00") ->
(db:open-pre("00", 0), ...)
- apply text index for "urn:123"
- rewrite cached filter to cached path: ((db:open-pre("00", 0),
...))[(descendant::element(lmerFile:format)/text() = "urn:123... -> db:text("00",
"urn:123")/parent::element(lmerFile:format)/ancest...
- rewrite where clause(s)
- simplify FLWOR expression: db:text("00",
"urn:123")/parent::element(lmerFile:format)/ancest...
Optimized Query:
count((# Q{http://basex.org/modules/db}enforceindex #) { db:text("00",
"urn:123")/parent::element(lmerFile:format)/ancestor::node()[self::document-node()] })
Query:
declare namespace lmerFile = "http://www.ddb.de/LMERfile;; count( (# db:enforceindex #) {
let $dbs := ("00") for $db in $dbs for $doc in db:open($db) where
$doc//lmerFile:format/text() = 'urn:123' return $doc })
Result:
- Hit(s): 1 Item
- Updated: 0 Items
- Printed: 4 b
- Read Locking: (global)
- Write Locking: (none)
Timing:
- Parsing: 0.26 ms
- Compiling: 55.69 ms
- Evaluating: 60.66 ms
- Printing: 0.37 ms
- Total Time: 116.98 ms
Query Plan:
db:enforceindex
urn:123
[2]
Compiling:
- pre-evaluate expression list to xs:string sequence: ("00", "01")
- merge steps: descendant::element(lmerFile:format)
- inline $dbs_0
- rewrite where clause(s)
Optimized Query:
count((# Q{http://basex.org/modules/db}enforceindex #) { for $db_1 in ("00", "01") return
(db:open($db_1))[(descendant::element(lmerFile:format)/text() = "urn:123")] })
Query:
declare namespace lmerFile = "http://www.ddb.de/LMERfile;; count( (# db:enforceindex #) { let $dbs
:= ("00", "01") for $db in $dbs for $doc in db:open($db) where
$doc//lmerFile:format/text() = 'urn:123' return $doc })
Result:
- Hit(s): 1 Item
- Updated: 0 Items
- Printed: 4 b
- Read Locking: (global)
- Write Locking: (none)
Timing:
- Parsing: 0.27 ms
- Compiling: 0.54 ms
- Evaluating: 1778.21 ms
- Printing: 0.31 ms
- Total Time: 1779.33 ms
Query Plan:
db:enforceindex
00
01
urn:123
-Ursprüngliche Nachricht-
Von: Christian Grün
Gesendet: Montag, 13. Januar 2020 13:07
An: Braunschweig, Björn
Cc: basex-talk@mailman.uni-konstanz.de
Betreff: Re: [basex-talk] slow search performance across multiple databases
Dear Björn,
Our Article on Indexes may give you some hints [1]:
• Via the 'enforceindex' pragma, you can help the optimizer by indicating that
all of your database have up-to-date index structures.
• With db:text, you can directly access the database indexes and rewrite the
remaining steps of your query in reverse order:
db:text($db, 'urn:123')/parent::lmerFile:format/...
Hope this helps
Christian
[1] http://docs.basex.org/wiki/Indexes
On Mon, Jan 13, 2020 at 11:56 AM Braunschweig, Björn
wrote:
Hello everyone,
First of all thank you for building such a great product!
Unfortunately I encounter extremely slow queries using a basex
instance. My xml files are distributed over 256 databases (00-ff) with
roughly 250MB each. All share a structure similar to [3]. I executed
OPTIMIZE ALL for all databases, see [4]. [5] shows the info
Re: [basex-talk] Weird: mixed content trimmed unexpectedly
Ok so I remembered correctly. Can I pass this in a REST PUT operation? Am 09.12.2019 um 16:53 schrieb Martin Honnen: On 09.12.2019 16:51, Omar Siam wrote: Probably there is an option (CHOP?) that one can set to tell BaseX not to trim whitespace at the edges of text nodes. Yes, see http://docs.basex.org/wiki/Options#XML_Parsing
Re: [basex-talk] running in parallel
Hi,
I see the same in my application. My two cent of wisdom: I would say
most disks today will be fast enough to mask this problem. Let alone
SSDs that can happily fetch two files at the (almost) same time. But the
thing is: The exist code uses some pretty heavy locks to make sure no
two Java threads access the same (database) file at the same time. And
unless this is really given some thought for data safety I am glad that
it does not allow queries to run in parallel. I would love to solve this
in a more state of the art way but got burned in the past by multi
threading. So I have great respect for any good, safe and fast
implementation multi threading file access implementation. I fear no one
did one yet for BaseX.
Best regards
Omar Siam
Am 08.12.2019 um 17:04 schrieb Markus Wittenberg:
Hi Giuseppe,
as long as the files are not on physically different disks, you will
have the two functions block each other with read and write
operations. And BaseX runs lots of code in parallel without you
explicitly telling it so.
Best regards,
Markus
Am 08.12.2019 um 16:48 schrieb cel...@informatik.uni-leipzig.de:
Hi,
I am trying to run two BaseX scripts in parallel using:
xquery:fork-join(
(
function() {xquery:eval(xs:anyURI('extract_from_ocr1.xq'))}
,
function (){xquery:eval(xs:anyURI('extract_from_ocr2.xq'))}
)
)
As far as I can understand (read below), the scripts are kind of run
in parallel, but still the time benefit of that does not seem much in
comparison with running in sequence (~25s vs ~28s). The files contain
the same function, which reads files from a directory, performs some
calculation, and saves the result in a file (the two scripts work on
different directories). I infer that the previous script is run in
parallel because the files for the results are created at the same time.
I tried to do the same with GNU parallel, and in that case the files
are actually run in parallel.
Do we know why the execution time is not (more or less) halved in
BaseX? Thanks.
Ciao,
Giuseppe
Re: [basex-talk] Weird: mixed content trimmed unexpectedly
Hello, I had some expierence of this of my own. Saxon and other XML tools are better at guessing what the user wants. Probably there is an option (CHOP?) that one can set to tell BaseX not to trim whitespace at the edges of text nodes. But to my knowledge the standard way is to add @xml:space="preserve" to the outermost element where this is necessary. That's how I do it. test for me Best regards Omar Siam Am 09.12.2019 um 16:32 schrieb Arjan Loeffen: Dear BaseX People, after many (happy!) projects using BaseX I have found that curl -i -X PUT --basic --user admin ^ -H "Content-Type: application/xml" -d " test for me " "http://localhost:8984/rest/LeapinLists/test.xml; stores test forme To the database. Notice the spaces in the XML string. What may have caused this behavior? Any suggestions on what to do here? I tried it on Basex 9.3, the latest edition right now. -- *Arjan Loeffen* Armatiek BV & Armatiek Solutions BV 06-12918997
Re: [basex-talk] HTTPServer + gzip compression
Yes, it works! 1000 entries of XML in a JSON wrapper from my data are now transfered as 193 KB instead of 1.6 MB. Not bad. Unfortunately on localhost that doesn't make much of a difference in overall timing as measured in Chrome's development tools. For real internet connections I think this is quite an improvement. Best regards Omar Siam Am 19.11.2019 um 01:28 schrieb Christian Grün: The snapshot has been updated. Is it working now? On Mon, Nov 18, 2019 at 6:41 PM Omar Siam wrote: I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? Best regards Omar Siam
Re: [basex-talk] HTTPServer + gzip compression
I just wanted to try gzip http compression and downloaded the current build BaseX93-20191116. The option is not there anymore. What happened? Best regards Omar Siam
Re: [basex-talk] I am looking for the fastest way to sort 2.4 Mio tags by two attribute ascending and descending
is code
declare namespace _ = "https://www.oeaw.ac.at/acdh/tools/vle/util;;
let $all := collection("_qdb-TEI-02__cache")//_:dryed[@order='ascending'
and not(@label)]/tokenize(@ids)
return db:attribute("_qdb-TEI-02__cache", subsequence($all, 1000, 25))[.
instance of attribute(ID) or . instance of
attribute(xml:id)]/..!db:open-pre(./@db_name, ./@pre)
showing a page takes about 500 ms.
Best regads
Omar Siam
[basex-talk] I am looking for the fastest way to sort 2.4 Mio tags by two attribute ascending and descending
Hi,
I have a custom index that looks like this (one db, different files):
<_:dryed xmlns:_="https://www.oeaw.ac.at/acdh/tools/vle/util;
db_name="z881_qdb-TEI-02n" order="none">
<_:d pre="15627" db_name="z881_qdb-TEI-02n" xml:id="z881_qdbn-d16e2"
vutlsk="tsįttr Ziter [Subst]" vutlsk-archiv="HK 881, z8810118.sch#1"/>
<_:d pre="15673" db_name="z881_qdb-TEI-02n" xml:id="z881_qdbn-d16e21"
vutlsk="tsįttr Ziter [Subst]" vutlsk-archiv="HK 881, z8810118.sch#1"/>
...
<_:dryed xmlns:_="https://www.oeaw.ac.at/acdh/tools/vle/util;
db_name="f227_qdb-TEI-02n" order="none">
<_:d pre="467" db_name="f227_qdb-TEI-02n" xml:id="f237_qdb-d1e29398"
vutlsk="(aus)faren [Verb]" vutlsk-archiv="HK 327, f227#944.1 =
fare0126.eck#1.1"/>
<_:d pre="591" db_name="f227_qdb-TEI-02n" xml:id="f237_qdb-d1e29438"
vutlsk="(aus)faren [Verb]" vutlsk-archiv="HK 327, f227#945.1 =
fare0126.eck#2.1"/>
...
There are about 2.4 Mio _:d tags in this db.
I need to sort them by the @vutlsk* attributes alphabetically in
ascending and descending order.
With the code I have now:
declare namespace _ = "https://www.oeaw.ac.at/acdh/tools/vle/util;;
let $sorted-ascending := subsequence(for $d in
collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d
order by $d/@vutlsk ascending
return $d/(@ID, @xml:id)/data(), 1, 1)
let $sorted-descending := subsequence(for $d in
collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d
order by $d/@vutlsk descending
return $d/(@ID, @xml:id)/data(), 1, 1)
let $sorted-ascending-archiv := subsequence(for $d in
collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d
order by $d/@vutlsk-archiv ascending
return $d/(@ID, @xml:id)/data(), 1, 1)
let $sorted-descending-archiv := subsequence(for $d in
collection('_qdb-TEI-02__cache')//*[@order="none"]/_:d
order by $d/@vutlsk-archiv descending
return $d/(@ID, @xml:id)/data(), 1, 1)
return (db:replace("_qdb-TEI-02__cache", 'ascending_cache.xml', <_:dryed
order="ascending" ids="{string-join($sorted-ascending, ' ')}"/>),
db:replace("_qdb-TEI-02__cache", 'descending_cache.xml', <_:dryed
order="descending" ids="{string-join($sorted-descending, ' ')}"/>),
db:replace("_qdb-TEI-02__cache", 'ascending-archiv_cache.xml', <_:dryed
order="ascending" label="archiv"
ids="{string-join($sorted-ascending-archiv, ' ')}"/>),
db:replace("_qdb-TEI-02__cache", 'descending-archiv_cache.xml', <_:dryed
order="descending" label="archiv"
ids="{string-join($sorted-descending-archiv, ' ')}"/>))
This takes 30 s to about a minute depending on the subsequence I choose.
I did experiments with doing multithreading and not. Multiple jobs or
fork-join make it worse.
Worst case I need to do it every time I save a change to the original
DBs for which I maintain that index.
Any ideas how to speed this up?
Best regards
Omar Siam
Re: [basex-talk] Serialization mismatch using Saxon for xslt:transform
Hi Steve! I take from these results that the output from Saxon xslt:transform is serialized according to the stylesheet, and then parsed again by BaseX as xml on the way to being serialized again on output from the function, and the error is coming from that implicit parse. The communication between BaseX (and others) and Saxon is: * BaseX passes XML (serailizes it or passes some object which Saxon uses to read the document into its own internal representation) * Saxon genereates an output according what you configure using * BaseX has to read and interpret that output to process it any further. The last step means that Saxon has to generate something that BaseX can consume and that is XML most of the time. method html on purpose uses some constructs that are not wellformed XML and uses entitiies always defined in HTML to be more compatible with some now outdated browsers. In the end because you run Saxon to transform your XML and then (possibly) process it using XQuery again to generate output BaseX is the tool that has to be told what the output should look like. Saxon has to be used in a way that BaseX unterstands. So in the end you will most of the times end up with having to tell Saxon to produce XML or XHTML. You may be able to do this using two XSL stylesheets that only contain and import the actual stylesheet. I tried xslt:transform-text() but it escapes all of the element tags. I didn't try that but as I understand it xslt:transform-text() should give you some unparsed text that BaseX can't process any further but if output by BaseX (using a text method from BaseX standpoint probably) should be the short circuit you are looking for. Best regards Omar
[basex-talk] Activating gzip compression
Hi, I tried both ways of enabling gzip compression for JSON and js (and others). I can't see it actually working. The GzipFilter method from the web.xml sample seems to be deprecated with a warning of "it is bug ridden" ... But then in a browser debugger I cannot see it actually zipping anything. I trief to enable this in my BaseX 9.1.2 instance using GzipFilter org.eclipse.jetty.servlets.GzipFilter mimeTypes text/html,text/xml,text/ecmascript,application/xhtml+xml,application/json,application/javascript,image/svg+xml GzipFilter /* The GzipHandler method seems to get loaded but then also in a browser debugger I cannot see it actually zipping anything. For the GzipHandler the documentation https://www.eclipse.org/jetty/documentation/current/gzip-filter.html talks about when it actually gets invoked and mentions: * No content-encoding is specified by the resource I found no way to not specify an encoding for whatever I return in RESTXQ. If I don't specify anything it's UTF-8. it shows up as "; character-set=UTF-8"I did not find a way to specify no encoding. Maybe unrelated but perhaps that is a problem. I modified webapp/WEB-INF/jetty.xml like this so it uses the GzipHandler (copied from the current jetty distribution): ... I hope there is a way to get this to work. Best regards Omar Siam
Re: [basex-talk] Using HTTP Digest Authentication in a RESTXQ services
Hi,
Looks likt I got BaseX 9.1.2 to work.
I configured the AUTHMETHOD in .basex to Digest
In the %perm:check funktion I return 401 if I want authentication. As
noted in commit
https://github.com/BaseXdb/basex/commit/acd4713e99f618c8dc9dda8d5f994d0139c2bf42
for 9.2 there is no way to send a custom WWW-Authentication header but
this triggers the BaseX/jetty built in authentication, be that Basic,
Digest or "Custom"
So if I return a response containing this part:
{(: Never before 9.2 used
see AUTHMETHOD --> :)}
I can decide in code when i need a user to autheticate.
Hope that keeps working.
Best regards
Omar
Am 15.02.2019 um 16:28 schrieb Christian Grün:
Hi Omar,
As I see it there is no way of doing this selectively.
Right, there is no such solution at the moment. We didn’t implement
digest authentication via our permission layer so far, but it should
definitely be possible (might be an interesting challenge… any
volunteers out there?).
Best,
Christian
Re: [basex-talk] Using HTTP Digest Authentication in a RESTXQ services
Hi and thanks for the quick response! Ok. Now I see how that works. As I see it there is no way of doing this selectively. Like having the / page not having authentication and /dba do authentication using some other method. Or if I have some service where I need to decide at runtime in XQuery code that I now would like to require the client to do digest authentication. Digest authentication cannot be hooked to the %perm annotation I think. Is that correct or am I missing something? Best regards Omar Am 14.02.2019 um 18:39 schrieb Christian Grün: Hi Omar, Digest authentication with RESTXQ should work out of the box. The following changes in the configuration may be required: • You’ll need to enable digest authentication in the web.xml configuration file [1] • By default, the admin user is enabled for RESTXQ requests. You’ll need to remove the corresponding entry [2] Best Christian [1] https://github.com/BaseXdb/basex/blob/9a089491894a01f9d8edb5d30e695f18b13bc9c4/basex-api/src/main/webapp/WEB-INF/web.xml#L93-L96 [2] https://github.com/BaseXdb/basex/blob/9a089491894a01f9d8edb5d30e695f18b13bc9c4/basex-api/src/main/webapp/WEB-INF/web.xml#L32-L35 On Thu, Feb 14, 2019 at 6:35 PM Omar Siam wrote: Hi list! I just experimented a bit to see if there is an obvious way to use HTTP digest authentication for some RESTXQ service I plan to write. I would prefer not to reimplement the HTTP digest logic in XQuery. I thought maybe there is a way to use the built in digest logic like for the REST endpoint. I don't seem to get that to work. Any suggestions? Best regards Omar Siam
[basex-talk] Using HTTP Digest Authentication in a RESTXQ services
Hi list! I just experimented a bit to see if there is an obvious way to use HTTP digest authentication for some RESTXQ service I plan to write. I would prefer not to reimplement the HTTP digest logic in XQuery. I thought maybe there is a way to use the built in digest logic like for the REST endpoint. I don't seem to get that to work. Any suggestions? Best regards Omar Siam
Re: [basex-talk] Different interpretation of regex in eXist, Saxon and BaseX
Sorry I got that wrong. I meant XQuery has greedy (the default) and reluctant. But not possessive.
Re: [basex-talk] Different interpretation of regex in eXist, Saxon and BaseX
Hi! My point was that greediness is *not* part of the XQuery RegExp standard. Java on the other hand has this feature: https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#greedy and others. And I don't know about Perl, PHP, Python and so on. What I want to stress is: A beautiful RegExp from the internet may or may not work with a particular RegExp implementation. Nevertheless as Saxon is well integrated in BaseX you can use it to do some RegExp work. Just getting data to and from Saxon may be not possible depending on the size of what you want to process. Saxon always works on a in-memory-representation of the data as far as I know and that is not an option with a 2.5 GB XML for example. Best regards Omar Am 09.08.2018 um 16:32 schrieb Andreas Mixich: Omar Siam wrote: Using the java regular expression implementation you can use greedy and some other things. The XSL and XQuery implementation according to the standards does not allow this and so misinterpretes the regular expression. See here: I checked https://www.w3.org/TR/xpath-functions-31/#regex-syntax and also the https://www.w3.org/TR/xmlschema-2/#regexs but did not find any mention of greediness. But then, I am not sure, whether I understood this from latter document: A ·regular expression· R is a sequence of characters that denote a set of strings L(R). When used to constrain a ·lexical space·, a regular expression R asserts that only strings in L(R) are valid literals for values of that type. For all ·atom·s S and non-negative integers n, m such that n <= m, valid ·piece·s R are: Denoting the set of strings L(R) containing: S? the empty string, and all strings in L(S). Now I am not quite sure what L(S) means. You can tell Saxon to use a different regexp engine such as the standard Java one: https://www.saxonica.com/html/documentation/functions/fn/matches.html The hint is much appreciated, though BaseX is my actual development target. I just mentioned Saxon and eXist, because I cross checked them and found the result to be interesting enough to be taken to the list (and still hope, that Christian chimes in and may find a good reason, to do it the other way around in opposition to the way it is now)
Re: [basex-talk] Different interpretation of regex in eXist, Saxon and BaseX
Hi I think the problem is: There are numerous implemetations of regular expressions which have a common subset but are different in the more advanced features. Using the java regular expression implementation you can use greedy and some other things. The XSL and XQuery implementation according to the standards does not allow this and so misinterpretes the regular expression. See here: https://www.w3.org/TR/xpath-functions-31/#regex-syntax You can tell Saxon to use a different regexp engine such as the standard Java one: https://www.saxonica.com/html/documentation/functions/fn/matches.html Best regards Omar Am 07.08.2018 um 21:38 schrieb Andreas Mixich: Hi [rfc3986](https://tools.ietf.org/html/rfc3986#appendix-B) defines a nice regular expression, which groups any URI, including URN, by URI component. Interesting about this regex is the use of the '?' quantifier which makes every preceding group/component optional, thus matching either an URI or any other(!) string, since anything, that does not match one of the special groups, goes into a catch-all group (no.5), which keeps either the path or the full, arbitrary string. This is neglectable, since the input to this regex is guaranteed to be of the right type (a/@href/string()). Here is the relevant part from the RFC. Appendix B ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))? 123 4 5 6 78 9 The numbers in the second line above are only to assist readability; they indicate the reference points for each subexpression (i.e., each paired parenthesis). We refer to the value matched for subexpression as $. For example, matching the above expression to http://www.ics.uci.edu/pub/ietf/uri/#Related results in the following subexpression matches: $1 = http: $2 = http $3 = //www.ics.uci.edu $4 = www.ics.uci.edu $5 = /pub/ietf/uri/ $6 = $7 = $8 = #Related $9 = Related where indicates that the component is not present, as is the case for the query component in the above example. Therefore, we can determine the value of the five components as scheme= $2 authority = $4 path = $5 query = $7 fragment = $9 Going in the opposite direction, we can recreate a URI reference from its components by using the algorithm of Section 5.3. I tested this regex with Saxon, eXist and BaseX. eXist successfully parsed all the test-cases, I threw at it, into the right groups, Saxon and BaseX did not. The failure is: [FORX0003] Pattern matches empty string.. And that got me baffled, since all three processors use Java underneath and since the definition of the '?' quantifier, when used like this, seems to be: Makes the preceding item optional. Greedy, so the optional item is included in the match if possible. Which means, that *if* any of the group's contents match, they should be included, rather than producing an empty string. Why is it like that? And what can I do about it? I found no other URI parsing regex, that componentizes this way and would be compatible with XQuery. See, attached, a test-case.
Re: [basex-talk] BaseX and oXygen
Hi Andreas, It is still be possible. Execution is all about the XQJ connection and a transformation scenario using it. Just tested it with my setup in oXygen 20.1 and a BaseX 9.0.2 server running as backend. The problem is: This is by design limited to XQuery 1.0. No maps, no arrays, no string concatenation operator and other things missing. There may be "old style" function equivalents but it is cumbersome. I stopped using oXygen for BaseX XQuery for those reasons. I'm happy with the GUI. Best regards Omar Siam Am 01.08.2018 um 23:16 schrieb Andreas Mixich: Hi, I remember, that I was able to run BaseX as execution engine for the oXygenXML XQuery editor. This seems to be impossible since a few oXygen versions. All that's left is the WebDAV part. I did not manage to run my old config, creating new ones did not help either. I remember having read a post by Michael Kay, who seemed to have some problems with XQJ not being up-to-date and some licensing issues making it impossible to fork/update or something like that. The bug-tracker and mailing-list links on the XQJ site are dead as well.
[basex-talk] Is it possible to set the root for resolving include and import in xslt:transform
Hi list! I have an XSL snippet that includes and imports some XSL files. I pass this to xslt:transform and the includes and imports are resolved using the working directory where basex was started from. On my server that was / and on my Windows development machine C:\Users\user\basex\bin. This was not what I expected. Quick way to see the effect: declare namespace xsl="http://www.w3.org/1999/XSL/Transform;; xslt:transform(, http://www.w3.org/1999/XSL/Transform; version="2.0"> select="static-base-uri()"/> ) Perhaps in a future version it may be possible to set that to the directory the XQuery runs from or to be able to just specify what it should be. Best regards Omar Siam
[basex-talk] DBA session timeout
Hi list! I just got asked if I can change the session timeout for DBA. I actually can't. I don't know where to look. Can anybody can give me a hint? Best regards Omar Siam
Re: [basex-talk] baseX vs ExistDB
I use BaseX'es jobs for this. Works but you have to be careful because you give up all the protections against dead locking or becoming unresponsive because no new jobs can be scheduled anymore so all HTTP communication stops like database administration for example. Best regards Omar Am 20.04.2018 um 15:57 schrieb Marco Lettere: On 20/04/2018 15:49, Christian Grün wrote: Hi Omar, Thank you (and everyone else) for sharing your experiences. * BaseX in comparison to exist-db turned out to be particularly bad at hosting multiple XQuery based applications like RestXQ endpoint in one instance. Definitely true; BaseX was not built for that. If you want to run multiple applications with a single web server, the recommended approach is to use the WAR distributions of BaseX and deploy each application as a separate servlet. Or use it as we do here. A new BaseX process for every application. A sort of very small application container or nano-service with the optimization that they all share the BaseX code and just use different restxq and possibly data folders. ;-) M.
Re: [basex-talk] baseX vs ExistDB
Hi Christian! Am 20.04.2018 um 15:49 schrieb Christian Grün: * BaseX is not as smart on recognizing when indices can be used in longer XQuery code. This one is interesting to hear, because we observed that users chose BaseX in the past exactly because of the index rewritings. Did you encounter these restrictions when working with multiple databases, or also with single instances? This is in comparison to exist-db which finds indexes in whatever collection they are configured deep down in serveral hundrets of lines of XQuery module code where the collection name is calculated somwhere else. You know from some previous exchange we had that this is not the strong suite of BaseX. But that is a feature where exist-db is exceptionally good. Just when writing code structured like it was good for exist-db I didn't get any rewrites and so no acceleration from BaseX indexes at all. When rewriting queries to contain the db name as string literal that is a different story. This is also needed for get out of global locking. Best regards Omar ACDH-OeAW
Re: [basex-talk] baseX vs ExistDB
Hi all! I am dealing with 730 XML files with about 2.5 GB in total size right now for some months in BaseX. I'm happy to share the knowledge I gathered. We also tried exist-db on the same set of XML data and couldn't do any updates anymore in a reasonable amount of time. The most positve aspects of BaseX in my scenario are * It is easy to understand what BaseX is doing and when * If you like you can manage your updates in a very granular way in parallel using jobs. This can speed up things quit a lot. * You may be able to devide your XML into multiple BaseX databases in one instance and then access and update them without having locking problems and with speed. * You decide if and when you recreate indices after updates. The downside is * If you start doing things in parallel you can run into all sorts of locking and memory management problems. Memory can also be an issue if you do updates all over the place in a single run because then the update log can get really big. Also of course you can make your development system stall because you use up all the CPU ;-) * BaseX in comparison to exist-db turned out to be particularly bad at hosting multiple XQuery based applications like RestXQ endpoint in one instance. It is really easy to have a global (write) lock. Then things get stuck. * BaseX is not as smart on recognizing when indices can be used in longer XQuery code. exist-db is definitly better at that. If one keeps it simple and one project per BaseX instance then it is much easier to know what actually happens compared to exist-db and that is a big asset for me. Best regards Omar Siam ACDH-OeAW Am 19.04.2018 um 17:26 schrieb Feargal Hogan: On 18 Apr 2018, at 21:12, Liam R. E. Quin <l...@w3.org> wrote: On Wed, 2018-04-18 at 14:39 +0100, Feargal Hogan wrote: Hi Is anyone aware of any comparisons between baseX and Exist? I have some familiarity with Exist and I’d like o understand what are the benefits of each. What really matters is suitability to task, though, and that will depend on what you're trying to do. And part of suitability to task is the support network - are other people doing similar thigns using eXist-db or BaseX? Liam Hmmm, havent seen anyone doing what I am looking to do. Initially, I want to replace filesystem storage for about 12k xml files with queryable storage. As we progress, I may want to batch update mutiple records contextually and/or enhance the xml based on regex patterns. From the comparison chart that Ben referenced earlier I noticed that baseX doesn’t seem to actually load xml files into an xml database, is that right? So what does it do then? It creates a queryable indexed representation of the files? Is that right? And what happens when a file is edited/updated? Does baseX need to be 'told' that it has been updated, in order to add the new data to its indeices? Or does it know there has been an update and automatically reindex? Thanks Feargal
[basex-talk] JTS Library homepage moved
Hi list! This has a new owner/website: https://projects.eclipse.org/projects/locationtech.jts If you have time please update the link Best regards Omar Siam
Re: [basex-talk] Dynamic xpath
As I do the same in my code:
* you can use xquery:eval as mentioned but:
** be careful how to pass in the db:open() or collection(). It is quite
easy to prevent BaseX from seeing that there are indexes and use them
** xquery:eval uses global locking. So no writing while such a query is
running. As I understand it as many queries as you like can read at the
same time.
* you can use jobs:eval with cache: true() and background the actual job
and have the option to lock only what you need but:
** you can dead lock between the background and the RestXQ request if
you are not careful
** there is a chance to accept to many RestXQ requests and so the actual
query is never processed (counter measer: Option FAIRLOCK)
** you have to check for unwanted statements using jobs:parse so you
catch updates if you don't want write access or starting more jobs in
that XPath/XQuery they pass
Both cases allow access to the full library of functions available in
exist (apart from updating ones in xquery:eval). One thing to be aware
about is the potential of abuse of httpclient:*.
Because I cannot use global locking and do have updates mixed with read
requests I have invested some time to get the jobs:eval right. But I'm
also not the seasoned BaseX developer yet.
Best regards
Omar
Am 02.02.2018 um 16:06 schrieb France Baril:
I'm trying to do something similar to this because I'll have to deal
with xpaths provided by end users as parameters to a rest query:
var xpath='div/p'
for $i in db:open('en-us'){$xpath}
return $i
I saw that it's possible to do it with other tools, but I didn't find
if it's possible with xquery and BaseX.
How SQL dbs do it:
SET @XML1='
OK
'
SET @PATH = '/Root[1]/Device[1]/Inspection[1]/Status[1]'
SELECT @XML1.query(@PATH)
Any suggestion that doesn't involve analyzing the query string and
trying to rebuild it in some way?
--
France Baril
Architecte documentaire / Documentation architect
france.ba...@architextus.com
Re: [basex-talk] Locking up BaseX RESTXQ using jobs
Maybe. But then again I want to use a very large amount of threads. I
read fairlock hurts there. And I actually like the idea of always having
a way to know the status of the job system. I will look into that.
Best regards
Omar Siam
Am 29.11.2017 um 16:30 schrieb Christian Grün:
Maybe it helps you to enable the FAIRLOCK option?
On Wed, Nov 29, 2017 at 4:24 PM, Omar Siam <omar.s...@oeaw.ac.at> wrote:
No, I don't think this is possible without the RESTXQ job waiting for a
result of another job it started itself.
I'm quite sure this is to be expected but then again it puzzeled me that it
is possible to send jobs to the RESTXQ endpoint so fast it is unable to
start it's slave script but just the RESTXQ jobs pile up.
I think it is a side effect of allowing any number of non locking jobs so
there is always a possibility to send a jobs:stop request or have a look at
the situation and get things on track again.
If you could make sure that jobs started by RESTXQ jobs get a chance to be
unqueued before accepting new RESTXQ jobs when the parallel limit is reached
would be a way to improve the situation. I'm thinking about a very little
perhaps increasing delay for accepting new RESTXQ jobs when the parallel
limit is reached. Just so queued jobs get a chance to start. Or a yield
function to explicitly give away some execution time so another (non RESTXQ)
job can execute.
Best regards
Omar Siam
Am 29.11.2017 um 16:07 schrieb Christian Grün:
Hi Omar,
jobs that are started via jobs:eval are independent from the original
job, but most probably the deadlocks occur because you are retrieving
the result in the same query (see the last comment in [1]). If you are
calling jobs:eval to get better performance, xquery:fork-join may be a
better choice.
Still, if you believe that the deadlock can also be triggered without
jobs:eval and jobs:result in the same query, we should definitely have
a look at this.
Hope this helps,
Christian
[1] http://docs.basex.org/wiki/Jobs_Module#jobs:result
On Wed, Nov 29, 2017 at 3:56 PM, Omar Siam <omar.s...@oeaw.ac.at> wrote:
Slave scripts and jobs are scripts to me I generate on the fly to hard
code
the DB name so locking and optimzing know what they deal with.
Some of the code I'm working with right now (now including an error so
this
kind of lock up does not happen:
declare %private function wde:eval($query as xs:string, $bindings as
map(*)?, $jobName as xs:string, $dontCheckQuery as xs:boolean) as item()*
{
let $too-many-jobs := if (count(jobs:list()) >=
xs:integer(db:system()//parallel)) then
error(xs:QName('wde:too-many-parallel-requests'), 'Too many parallel
requests!') else (),
$query-is-sane := $dontCheckQuery or wde:query-is-sane($query),
$j := jobs:eval($query, $bindings, map {
'cache': true(),
'id': 'wde:'||$jobName||'-'||jobs:current(),
'base-uri': $wde:basePath||'/wde_'||$jobName||'.xq'}), $_ :=
jobs:wait($j)
return jobs:result($j)
};
declare
%rest:path("wde/{$dict}/{$index}")
%rest:query-param("q", "{$q}")
%rest:query-param("type", "{$type}")
%rest:GET
function wde:query-index($dict as xs:string, $index as xs:string, $q as
xs:string*, $type as xs:string*) {
let $query-index-script := ``[
import module namespace wde = "http://acdh.oeaw.ac.at/wde; at 'rest.xqm';
declare variable $q external;
declare variable $index external;
declare variable $type external;
wde:do-query-index(collection("`{$dict}`"), $index, $q, $type)
]``
return wde:eval($query-index-script, map {
'q': $q,
'index': $index,
'type': $type
}, 'query-index', true())
};
And the last part you might need is something that is faster than a human
when it comes to start requests like node with a bunch of
superagent/supertest request promises:
Promise.all(requests)
.then(docs => {})
14:24:56.545127.0.0.1:52592 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546127.0.0.1:52593 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546127.0.0.1:52595 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546127.0.0.1:52594 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546127.0.0.1:52596 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.547127.0.0.1:52597 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.547127.0.0.1:52598 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.548127.0.0.1:52599 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.548127.0.0.1:52600 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.549127.0.0.1:52601 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.549127.0.0.1:52602 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.550127.0.0.1:52603 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.551127.0.0.1:52604 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.551127.0.0.1:52605 admin REQUEST [GET]
http:
Re: [basex-talk] Locking up BaseX RESTXQ using jobs
No, I don't think this is possible without the RESTXQ job waiting for a
result of another job it started itself.
I'm quite sure this is to be expected but then again it puzzeled me that
it is possible to send jobs to the RESTXQ endpoint so fast it is unable
to start it's slave script but just the RESTXQ jobs pile up.
I think it is a side effect of allowing any number of non locking jobs
so there is always a possibility to send a jobs:stop request or have a
look at the situation and get things on track again.
If you could make sure that jobs started by RESTXQ jobs get a chance to
be unqueued before accepting new RESTXQ jobs when the parallel limit is
reached would be a way to improve the situation. I'm thinking about a
very little perhaps increasing delay for accepting new RESTXQ jobs when
the parallel limit is reached. Just so queued jobs get a chance to
start. Or a yield function to explicitly give away some execution time
so another (non RESTXQ) job can execute.
Best regards
Omar Siam
Am 29.11.2017 um 16:07 schrieb Christian Grün:
Hi Omar,
jobs that are started via jobs:eval are independent from the original
job, but most probably the deadlocks occur because you are retrieving
the result in the same query (see the last comment in [1]). If you are
calling jobs:eval to get better performance, xquery:fork-join may be a
better choice.
Still, if you believe that the deadlock can also be triggered without
jobs:eval and jobs:result in the same query, we should definitely have
a look at this.
Hope this helps,
Christian
[1] http://docs.basex.org/wiki/Jobs_Module#jobs:result
On Wed, Nov 29, 2017 at 3:56 PM, Omar Siam <omar.s...@oeaw.ac.at> wrote:
Slave scripts and jobs are scripts to me I generate on the fly to hard code
the DB name so locking and optimzing know what they deal with.
Some of the code I'm working with right now (now including an error so this
kind of lock up does not happen:
declare %private function wde:eval($query as xs:string, $bindings as
map(*)?, $jobName as xs:string, $dontCheckQuery as xs:boolean) as item()* {
let $too-many-jobs := if (count(jobs:list()) >=
xs:integer(db:system()//parallel)) then
error(xs:QName('wde:too-many-parallel-requests'), 'Too many parallel
requests!') else (),
$query-is-sane := $dontCheckQuery or wde:query-is-sane($query),
$j := jobs:eval($query, $bindings, map {
'cache': true(),
'id': 'wde:'||$jobName||'-'||jobs:current(),
'base-uri': $wde:basePath||'/wde_'||$jobName||'.xq'}), $_ := jobs:wait($j)
return jobs:result($j)
};
declare
%rest:path("wde/{$dict}/{$index}")
%rest:query-param("q", "{$q}")
%rest:query-param("type", "{$type}")
%rest:GET
function wde:query-index($dict as xs:string, $index as xs:string, $q as
xs:string*, $type as xs:string*) {
let $query-index-script := ``[
import module namespace wde = "http://acdh.oeaw.ac.at/wde; at 'rest.xqm';
declare variable $q external;
declare variable $index external;
declare variable $type external;
wde:do-query-index(collection("`{$dict}`"), $index, $q, $type)
]``
return wde:eval($query-index-script, map {
'q': $q,
'index': $index,
'type': $type
}, 'query-index', true())
};
And the last part you might need is something that is faster than a human
when it comes to start requests like node with a bunch of
superagent/supertest request promises:
Promise.all(requests)
.then(docs => {})
14:24:56.545127.0.0.1:52592 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546127.0.0.1:52593 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546127.0.0.1:52595 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546127.0.0.1:52594 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546127.0.0.1:52596 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.547127.0.0.1:52597 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.547127.0.0.1:52598 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.548127.0.0.1:52599 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.548127.0.0.1:52600 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.549127.0.0.1:52601 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.549127.0.0.1:52602 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.550127.0.0.1:52603 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.551127.0.0.1:52604 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.551127.0.0.1:52605 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.552127.0.0.1:52606 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.552127.0.0.1:52607 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.552127.0.0.1:52608 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.554127.0.0.1:52609 admin REQUEST [GET]
http://localhost:8984/wde/...
Best regards
Omar Siam
Am 29.11.2017 um 1
Re: [basex-talk] Locking up BaseX RESTXQ using jobs
Slave scripts and jobs are scripts to me I generate on the fly to hard
code the DB name so locking and optimzing know what they deal with.
Some of the code I'm working with right now (now including an error so
this kind of lock up does not happen:
declare%private functionwde:eval($queryasxs:string, $bindingsasmap(*)?,
$jobNameasxs:string, $dontCheckQueryasxs:boolean) asitem()*{
let$too-many-jobs:=if(count(jobs:list())
>=xs:integer(db:system()//parallel)) then
error(xs:QName('wde:too-many-parallel-requests'),'Too many parallel
requests!') else(),
$query-is-sane:=$dontCheckQueryorwde:query-is-sane($query),
$j:=jobs:eval($query, $bindings, map {
'cache': true(),
'id':'wde:'||$jobName||'-'||jobs:current(),
'base-uri': $wde:basePath||'/wde_'||$jobName||'.xq'}), $_:=jobs:wait($j)
returnjobs:result($j)
};
declare
%rest:path("wde/{$dict}/{$index}")
%rest:query-param("q","{$q}")
%rest:query-param("type","{$type}")
%rest:GET
functionwde:query-index($dictasxs:string, $indexasxs:string,
$qasxs:string*, $typeasxs:string*) {
let$query-index-script:=``[
importmodulenamespacewde ="http://acdh.oeaw.ac.at/wde"at'rest.xqm';
declarevariable$qexternal;
declarevariable$indexexternal;
declarevariable$typeexternal;
wde:do-query-index(collection("`{$dict}`"), $index, $q, $type)
]``
returnwde:eval($query-index-script, map {
'q': $q,
'index': $index,
'type': $type
},'query-index', true())
};
And the last part you might need is something that is faster than a
human when it comes to start requests like node with a bunch of
superagent/supertest request promises:
Promise.all(requests)
.then(docs=>{})
14:24:56.545 127.0.0.1:52592 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546 127.0.0.1:52593 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546 127.0.0.1:52595 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546 127.0.0.1:52594 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.546 127.0.0.1:52596 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.547 127.0.0.1:52597 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.547 127.0.0.1:52598 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.548 127.0.0.1:52599 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.548 127.0.0.1:52600 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.549 127.0.0.1:52601 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.549 127.0.0.1:52602 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.550 127.0.0.1:52603 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.551 127.0.0.1:52604 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.551 127.0.0.1:52605 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.552 127.0.0.1:52606 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.552 127.0.0.1:52607 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.552 127.0.0.1:52608 admin REQUEST [GET]
http://localhost:8984/wde/...
14:24:56.554 127.0.0.1:52609 admin REQUEST [GET]
http://localhost:8984/wde/...
Best regards
Omar Siam
Am 29.11.2017 um 15:42 schrieb Christian Grün:
Hi Omar,
Thanks for reporting your observation.
What do you exactly mean by ”slave jobs”? Do you have some guidance
for us to make this reproducible?
Best,
Christian
On Wed, Nov 29, 2017 at 3:20 PM, Omar Siam <omar.s...@oeaw.ac.at> wrote:
Hi!
I just observed the following: If I start more RESTXQ requests than the
PARALLEL settings allows jobs, which start slave jobs and wait for them, the
RESTXQ jobs never finish, hang.
The RESTXQ jobs don't lock anything but start a slave job each which does
lock some DB for reading.
It seems to me that the RESTXQ requests are started regardless of the number
of parallel jobs already running but the slave jobs never start because they
are over the limit of parallel running jobs.
Is this a known situation? Is there some way to stop accepting RESTXQ
requests or not starting slave jobs until the parallel limit permits it?
Best regards
Omar Siam
[basex-talk] Locking up BaseX RESTXQ using jobs
Hi! I just observed the following: If I start more RESTXQ requests than the PARALLEL settings allows jobs, which start slave jobs and wait for them, the RESTXQ jobs never finish, hang. The RESTXQ jobs don't lock anything but start a slave job each which does lock some DB for reading. It seems to me that the RESTXQ requests are started regardless of the number of parallel jobs already running but the slave jobs never start because they are over the limit of parallel running jobs. Is this a known situation? Is there some way to stop accepting RESTXQ requests or not starting slave jobs until the parallel limit permits it? Best regards Omar Siam
Re: [basex-talk] xsl - using doc() to read in an xml stored within BaseX
I prefer using a RESTXQ service that exposes exactly what you want and only that. By default like this you need no password but you should check that you don't open up your documents to the world if this is an issue for you. Just don't try that with BaseX < 8.6.4. It just locks up. You could also pass the documents as parameters. It think that is pretty efficient. Best regards Omar Siam Am 24.11.2017 um 16:24 schrieb oliver.zmo...@degruyter.com: Dear all, currently I’m facing following problem: 1)Saxon9he.jar stored in “lib” folder (to ensure XSLT 2.0) of a local BaseX instance (Windows) 2)Xquery calls a transformation xslt:transform($doc, $xsl) 3)Within this stylesheet I try to address another resource stored in the same database Background: XML contains product information and refers to certain other XML resources like authors, serials etc. I guess I have to options: 1)Refer to document directly (database path) Here I used doc() and referred to the additional documents, e.g. doc(‘/serials/serial-123.xml’). Unfortunately, the transformation says that the file or directory does not exist. I played a while around with doc() and collection() but could not find a way to make it work. Maybe the reason is that saxon runs outside the database and does not know about the database context. 2)Use rest service Afterwards I tried to use the rest interface. Given the example before I used http://admin:admin@localhost:8984/rest/serials/serial-123.xml (user and pw are admin). Checked that simple with a browser – worked. The transformation instead didn’t work – it returned a 401 unauthorized http response. So, my question is how to read in database documents in an XSL stylesheet? And why does my http request does not work? Best, Oliver
[basex-talk] Hypertext Application Language: Do we need a library that implements this?
Hi list! I implemented a REST API using apigility some time ago and by doing that I stumbled upon http://stateless.co/hal_specification.html I kind of like it so I thought maybe there is a customizable XQuery implementation? If not: Is someone else interested in creating a customizable BaseX XQuery implementation? Best regards Omar Siam
Re: [basex-talk] DB locking
Hi,
As I'm working around the same sort of problem:
* I check for every xquery:eval, xquery:update and make it a jobs:eval,
they have to cause a global read or write lock respectively.
* I put every db:* (e. g. db:list()) in the code into it's own short
running job
* I replace every doc($var) and collection($var) with a
doc("stringconst") or collection("stringconst") (for example by using
the bxcode code for binding external db name variables mentioned earlier
on this list)
* I split my large DB consisting of about 700 XML documents with 2.5 GB
of data into 700 databases with about 5MB of data each.
* I stopped using DBA modules Queries interface because this implicitly
uses xquery:eval or xquery:update and so always leads to a global read
or write lock.
This will open a lot of opportunities for read queries to see a somewhat
inconsistent state but I'm willing to accept this.
So now I am down to a few seconds or less of locking for any particular
db. I hope this will enable me to do updates while not blocking read
queries for a longer period of time. I'm just working on a REST based
test for this.
Best regards
Omar Siam
Am 09.11.2017 um 04:19 schrieb Navin Rawat:
Hi Christine,
It is global lock. How can I prevent it? Should I need to avoid access
database while running jobs. It will work, or there is something else.
Regards,
Navin
On 08-Nov-2017 9:33 PM, "Christian Grün" <christian.gr...@gmail.com
<mailto:christian.gr...@gmail.com>> wrote:
Hi Navin,
> Is this the correct behavior of BaseX to block whole database
(even for
> reading)? or I am doing something wrong? Please suggest.
This depends on your queries (see [1]). You can use
jobs:list-details() to check if your jobs lead to local or global
locks.
Best,
Christian
[1] http://docs.basex.org/wiki/Transaction_Management
<http://docs.basex.org/wiki/Transaction_Management>
[basex-talk] Thanks for Enforce Index Rewriting!
This sounds like a great improvement for our code: http://docs.basex.org/wiki/Indexes#Enforce_Rewritings. Many thanks!
Re: [basex-talk] Jobs, locking and doing large updates
Am 18.10.2017 um 17:19 schrieb Christian Grün: Hi Omar, Beforehand: You mentioned that your RESTXQ always cause global locks. Do you have an example for that (see my last mail)? Sorry, no. While running that large update yesterday I thought I saw someone running a RESTXQ call without xquery:eval which was not processed because of the update in progress. But I could not reproduce it. I'll keep an eye on it but perhaps I just did not see the xquery:eval.
[basex-talk] Jobs, locking and doing large updates
Hi list! I was experimenting with the jobs module for the last weeks to speed up updates and to make them fit into less than 6 GB of memory. It does not work the way I expected. * Updating jobs don't seem to run in parallel even if they don't work with and lock the same database. Or do they start much later than I expected? * It seems to me there is an upper limit of jobs that can be queued (about 100?). I automatically started all of the update jobs but some updates did not run. The upside is: I can run my updates with just about 1 GB of memory which is much better for me. I work with dictionary like XML documents. They look like Contents with further tags Contents with further tags Contents with further tags ... a few thousand more ... Contents with further tags Contents with further tags I add or change larger parts of them and I also need to keep track of changes. So in a separate database there are old versions of entries with time stamps (@dt) like Contents with further tags
[basex-talk] Every query executed in dba Queries causes a global read lock
Hi list! Just discovered sth. unexpected: Every query executed in dba Queries causes a global read lock. Also other (very simple) queries using RESTXQ which use a DB explicit in the query are executed with a global read lock. Is this intentional? Is this just a side effect? I noticed that internally the code I run is wrapped in an xquery:eval. Maybe this causes this lock? Or does every RESTXQ cause a global read lock? I ran into this because I wanted to execute async code using this interface using the jobs module and then wait for the result. This of course locks up as soon as any access to data in dbs is attempted. Best regards Omar Siam
Re: [basex-talk] Is there any documentation on the narrow limits of XQuery index optimization in BaseX?
Hi!
Interesting ideas. I don't like the pragma idea that much because there
is already sth. like that with xquery:eval. The thing I miss most is a
function like xquery:eval that accepts a function as an argument but
also takes a context and does that runtime optimization. Or a way to
convert a function to string. Is there already sth. like this? I though
it might be xquery:invoke but that seems to do sth. else.
Best regards
Omar
Am 18.09.2017 um 15:59 schrieb Christian Grün:
Hi Omar,
Our current XQuery optimizer opens the addressed database in order to
find out if it has the required index structures, and if these are
up-to-date. Moreover, the cheapest index lookup will be selected if
there are several index candidates. For example, in the following
query, it will be likely that the second predicate will be rewritten
for index access:
db:open('persons')//person[country = 'Italy'][@id = 'id124']
If the addressed database is not statically known, these checks cannot
be performed that easily. Further implications and in-depth
information can be found in »Storing and Querying Large XML Instances«
[1].
Here are two ideas how this could be tackled:
• We could add an XQuery pragma to enforce specific index rewritings. Examples:
for $n in 1 to 10
for $db in ('persons' || $n)
for $person in db:open($db)//person
where (# basex:index #) { $person/country = 'Italy' }
where $person/@id = 'id124'
return $person
(1 to 10) ! db:open('persons' || .)//person
[(# basex:index #) { country = 'Italy' }]
[@id = 'id124']
• We could create multiple query plans at compile time (with and
without index, one rewriting for each index candidate) and choose the
one that is expected to be the cheapest at evaluation time. This would
definitely be the most flexible option (but the number of query plans
increases exponentially if you have nested FLWOR expressions and
queries with numerous predicates or where clauses).
Cheers,
Christian
[1] http://basex.org/about-us/publications/
[basex-talk] Is there any documentation on the narrow limits of XQuery index optimization in BaseX?
Hi list!
Recently I started to wonder why functions in my XQuery modules make no
use of indexes unless I force them to by using the respective function
like db:text(). Now I just did some I think minimal changes to the
example for text index at: http://docs.basex.org/wiki/Indexes#Text_Index.
If I just change it like below I loose almost all index optimization.
The only optimizations left are:
* Using db:open or collection with a string but not one in a variable of
a for FLOWR expression or the simple map operator.
* Using xquery:eval and passing a context with db:open or collection
Is there any chance that this will change any time soon? Is this a
fundamental restriction?
Best regards
Omar
xquery version "3.1";
declare namespace _ = "urn:local:namespace:_";
import module namespace functx = "http://www.functx.com;;
declare function _:_1st_example($ctx as document-node()) {
$ctx//*[text() = 'Germany']
};
declare function _:_2nd_example($file as xs:string) {
doc($file)//name[. = 'Germany']
};
declare function _:_3rd_example($dbname as xs:string+) {
(
for $c in ($dbname!collection(.))//country
where $c//city/name = 'Hanoi'
return $c/name,
$dbname!xquery:eval("//*[text() = 'Vietnam']", map {'': db:open(.)}))
};
(
(: 1st example :)
_:_1st_example(.),
(: 2nd example :)
_:_2nd_example('factbook.xml'),
(: 3rd example :)
_:_3rd_example(('factbook', 'factbook')),
xs:string(_:_1st_example)
)
Optimized Query:
(let $ctx_314 := . return $ctx_314/descendant::*[(text() = "Germany")],
db:open-pre("factbook",0)/descendant::name[(. = "Germany")], (for $c_317
in (("factbook", "factbook") !
collection(.))/descendant::country[(descendant::city/name = "Hanoi")]
return $c_317/name, (("factbook", "factbook") ! xquery:eval("//*[text()
= 'Vietnam']", map { "":db:open(.) }))), _:_1st_example cast as xs:string?)
Compiling:
[...]
- RUNTIME: pre-evaluate root() to document-node()
- RUNTIME: rewrite descendant-or-self step(s)
- RUNTIME: apply text index for "Vietnam"
- RUNTIME: pre-evaluate root() to document-node()
- RUNTIME: rewrite descendant-or-self step(s)
- RUNTIME: apply text index for "Vietnam"
Re: [basex-talk] Simple Map Operator in oXygen XML (XQJ)
Ok. Interesting. Please someone note this in the Wiki so there are no
surprises. For the ! syntax I can just stick with for in return but
missing map {} is a reason to switch to dba.
Best regards
Omar
Am 04.09.2017 um 17:14 schrieb Charles Foster:
Hi Michael,
BaseX XQJ supports XQuery 3.0 syntax I believe, but not XQuery 3.1
specific syntax.
It would be possible to update the XQJ driver to support XQuery 3.1
syntax, but it would mean upgrading the internal XQuery parser and
re-writer.
Omar, as a work around you could potentially use equivalent functions
in the standard F library (fn).
Kind Regards,
Charles
On 4 Sep 2017, at 22:02, Michael Seiferle <m...@basex.org
<mailto:m...@basex.org>> wrote:
Hi Omar,
/cc Hi @Charles,
you are right, this has nothing to with oXygen in particular!
Yet: I am not even sure if XQJ is meant to support anything but
XQuery 1.0.
Charles might correct me if I am wrong, but I don’t think there’s an
easy way to add 3.1 features to the current XQJ implementation.
Best from Konstanz
Michael
Am 04.09.2017 um 16:42 schrieb Omar Siam <omar.s...@oeaw.ac.at
<mailto:omar.s...@oeaw.ac.at>>:
Good idea but I was pretty sure that this comes from within BaseX
not oXygen. You can configure the parser and the executor of your
XQuery in oXygen und I used BaseX for both.
So I found in basex-examples a XQJ tutorial. And just by changing
Part1.java a little and filling in the example query from the Wiki I
also get
XQJQS001 - Invalid XQuery syntax, syntax does not pass static
validation.
Root Cause:
net.xqj.basex.bin.bB: Lexical error at line 1, column 12.
Encountered: " " (32), after : "!"
at
org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34)
at
org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34)
So most probably BaseX'es XQJ parser does not understand the simple
map operator.
XQSequence xqjs = xqje.executeQuery("xquery version \"3.1\"; (1 to 10) ! element
node { . }");
Am 04.09.2017 um 15:37 schrieb Michael Seiferle:
Hi Omar,
Looks like that error message is generated by oXygen; you might want to
crosspost to their list.
Best
Michael
Am 04.09.2017 um 14:31 schrieb Omar Siam<omar.s...@oeaw.ac.at>:
Hi!
I just tried to use ther Simple Map Operator while writing an XQuery in oXygen
XML and executing the query using a client/server BaseX 8.6.6 data source. I
configured it as described here:http://docs.basex.org/wiki/Integrating_oXygen
When I use the Simple Map Operator (!) I only get "Invalid XQuery syntax, syntax does not pass
static validation.". For example using "(1 to 10) ! element node { . }"
If I run that XQuery in DBA for example it works.
Any ideas?
Best regards
Omar Siam
Re: [basex-talk] Simple Map Operator in oXygen XML (XQJ)
Good idea but I was pretty sure that this comes from within BaseX not
oXygen. You can configure the parser and the executor of your XQuery in
oXygen und I used BaseX for both.
So I found in basex-examples a XQJ tutorial. And just by changing
Part1.java a little and filling in the example query from the Wiki I
also get
XQJQS001 - Invalid XQuery syntax, syntax does not pass static validation.
Root Cause:
net.xqj.basex.bin.bB: Lexical error at line 1, column 12.
Encountered: " " (32), after : "!"
at
org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34)
at
org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34)
So most probably BaseX'es XQJ parser does not understand the simple map
operator.
XQSequence xqjs = xqje.executeQuery("xquery version \"3.1\"; (1 to 10) ! element
node { . }");
Am 04.09.2017 um 15:37 schrieb Michael Seiferle:
Hi Omar,
Looks like that error message is generated by oXygen; you might want to
crosspost to their list.
Best
Michael
Am 04.09.2017 um 14:31 schrieb Omar Siam<omar.s...@oeaw.ac.at>:
Hi!
I just tried to use ther Simple Map Operator while writing an XQuery in oXygen
XML and executing the query using a client/server BaseX 8.6.6 data source. I
configured it as described here:http://docs.basex.org/wiki/Integrating_oXygen
When I use the Simple Map Operator (!) I only get "Invalid XQuery syntax, syntax does not pass
static validation.". For example using "(1 to 10) ! element node { . }"
If I run that XQuery in DBA for example it works.
Any ideas?
Best regards
Omar Siam
Re: [basex-talk] oXygen XML, content negotiation and #1220
Hi Christian,
Thanks for the answer. There is one thing I want to point out: My
problem is not the quality measure. I have problems because my client
lacks them. It sends 'application/xml,*/*' which is interpreted as: any
function that either satisfies application/xml or any other mime type.
Of course any othe mime type alone weill select more than one function
if there is more than one function specified for a path only
differentiated by mime/type in %rest:produces. The BaseX answer to this
is a BASX0003 error and therefore no meaningful result.
Although it is correct to assume */* does not really select any funtion
in particular I find that bahavior not helpful. I should be able to
specify what should be executed on */*.
Furthermore when some mime type is specified alongside */* I think that
*/* should be ignored and that mime type used instead.
Another solution would be to automatically assign a mime type more to
the left in the list a slightly higher quality measure so the problem of
a equal mime types can be solved. I even think there is some standard
about processing the Accept header in this manner but I may be wrong.
Best Regars
Omar
Am 04.07.2017 um 19:36 schrieb Christian Grün:
Hi Omar,
Thanks for the reminder. Yes, we still plan to improve support for
quality factors in our RESTXQ implementation. We’ll probably need to
find out how this has been resolved in JAX-RS, and define an
equivalent solution for RESTXQ. I have updated the initial comment in
Issue 1220 [1].
I am also passing this on to Adam Retter, the initiator of RESTXQ:
Adam, did you think about adding support for quality factors in the
RESTXQ implementation of eXist-db?
And, Omar, I assume you would like to have chosen the function that
matches the first content types given in the Accept header?
Thanks everyone,
Christian
[1] https://github.com/BaseXdb/basex/issues/1220
On Tue, Jul 4, 2017 at 1:17 PM, Omar Siam <omar.s...@oeaw.ac.at> wrote:
Hi list!
I like my oXygen editor to request raw XML data from my RESTxq endpoints to
test XSL stylesheets. When I do this with RESTxq paths that have different
%rest:produces it fails with the BASX0003 multiple functions error.
Turns out oXygen sends application/xml, */* as Accept header without any
quality.
I understand that as documented in
https://github.com/BaseXdb/basex/issues/1220 this is known not to work.
My/our workaround is to branch in a single XQuery function using sth. like:
if (some $a in tokenize(request:header("ACCEPT"), ',') satisfies $a =
('text/xml', 'application/xml')) then $xml else $xhtml
I hope this is still on the agenda for 8.7.
Best regards
Omar Siam
[basex-talk] oXygen XML, content negotiation and #1220
Hi list!
I like my oXygen editor to request raw XML data from my RESTxq endpoints
to test XSL stylesheets. When I do this with RESTxq paths that have
different %rest:produces it fails with the BASX0003 multiple functions
error.
Turns out oXygen sends application/xml, */* as Accept header without any
quality.
I understand that as documented in
https://github.com/BaseXdb/basex/issues/1220 this is known not to work.
My/our workaround is to branch in a single XQuery function using sth. like:
if (some $a in tokenize(request:header("ACCEPT"), ',') satisfies $a =
('text/xml', 'application/xml')) then $xml else $xhtml
I hope this is still on the agenda for 8.7.
Best regards
Omar Siam
