Re: [basex-talk] XML no longer auto formats?

[Bridger Dyson-Smith]

Hi Buddy -

The toggle for indenting xml results is here in the GUI (see attached;
second-to-last button on the right, in case the image doesn't make it
through). From the command line, `-W` should work for the standalone
executable.
Hope that's helpful.

Best,
Bridger



On Fri, Jun 28, 2024 at 7:31 PM Buddy Kresge 
wrote:

> Hello, one of my co-workers moved to BaseX 10.7 and the output that is
> displace when he runs his .xqm file is no longer pretty printed, where as
> in 9.7 the output does pretty print.
>
>
>
> Is there an option that can be turned on or off to pretty print the
> resultant XML?
>
>
>
> Buddy
>
>
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by Mimecast, a leader in email security and cyber
> resilience. Mimecast integrates email defenses with brand protection,
> security awareness training, web security, compliance and other essential
> capabilities. Mimecast helps protect large and small organizations from
> malicious activity, human error and technology failure; and to lead the
> movement toward building a more resilient world. To find out more, visit
> our website.
>

Re: [basex-talk] Issues with X11 display

[Bridger Dyson-Smith]

Hi Grayson

I'm not in front of a RHEL box, but have you tried the OpenJDK with a head?
Ie not the headless version.

Best,
Bridger

On Tue, Apr 30, 2024, 5:36 PM Graydon Saunders  wrote:

> Hello --
>
> After upgrading to Fedora 40, I find that I can not run the BaseX GUI.
> This is against BaseX110-20240426.163243.zip or 10.7
>
> 17:28 bin % ./basexgui
> /home/graydon/bin/basex/basex/.basex: writing new configuration file.
> Exception in thread "main" java.awt.HeadlessException:
> No X11 DISPLAY variable was set,
> or no headful library support was found,
> but this program performed an operation which requires it,
>
> at
> java.desktop/java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:166)
> at java.desktop/java.awt.Window.(Window.java:553)
> at java.desktop/java.awt.Frame.(Frame.java:428)
> at java.desktop/java.awt.Frame.(Frame.java:393)
> at java.desktop/javax.swing.JFrame.(JFrame.java:180)
> at org.basex.BaseXGUI.splash(BaseXGUI.java:96)
> at org.basex.BaseXGUI.(BaseXGUI.java:54)
> at org.basex.BaseXGUI.main(BaseXGUI.java:37)
>
>
> 17:23 bin % java -version
> openjdk version "21.0.2" 2024-01-16
> OpenJDK Runtime Environment (Red_Hat-21.0.2.0.13-4) (build 21.0.2+13)
> OpenJDK 64-Bit Server VM (Red_Hat-21.0.2.0.13-4) (build 21.0.2+13, mixed
> mode, sharing)
>
> Various attempts to set DISPLAY have not been helpful.
>
> Trying to use OpenJDK 17 produced effectively the same error message at
> greater length.
>
> Anyone got any suggestions on how to fix this?
>
> Thanks!
> Graydon
>

Re: [basex-talk] Out of Main Memory

[Bridger Dyson-Smith]

Hi Greg,

Have you tried experimenting with the ADDCACHE[1] option when building your
database? While it's been a bit, I recall having good results with,
especially in a RAM-constrained environment.
Hope that's helpful!
Best,
Bridger

[1] https://docs.basex.org/wiki/Options#ADDCACHE

On Thu, Mar 14, 2024 at 9:55 PM Murray, Gregory 
wrote:

> Thanks, Christian. I don’t think selective indexing is applicable in my
> use case, because I need to perform full-text searches on the entirety of
> each document. Each XML document represents a physical book that was
> digitized, and the structure of each document is essentially a header with
> metadata and a body with the OCR text of the book. The OCR text is split
> into pages, where one  element contains all the words from one
> corresponding printed page from the physical book. Obviously the number of
> words in each  varies widely based on the physical dimensions of the
> book and the typeface.
>
>
>
> So far, I have loaded 12,331 documents, containing a total of 2,196,771
> pages. The total size of those XML documents on disk is 4.7GB. But that is
> only a fraction of the total number of documents I want to load into BaseX.
> The total number is more like 160,000 documents. Assuming that the
> documents I’ve loaded so far are a representative sample, and I believe
> that’s true, then the total size of the XML documents on disk, prior to
> loading them into BaseX, would be about 4.7GB * 13 = 61.1GB.
>
>
>
> Normally the OCR text, once loaded, almost never changes. But the metadata
> fields do change as corrections are made. Also we add more XML documents
> routinely as we digitize more books over time. Therefore updates and
> additions are commonplace, such that keeping indexes up to date is
> important, to allow full-text searches to stay performant. I’m wondering if
> there are techniques for optimizing such quantities of text.
>
>
>
> Thanks,
>
> Greg
>
>
>
> *From: *Christian Grün 
> *Date: *Thursday, March 14, 2024 at 8:48 AM
> *To: *Murray, Gregory 
> *Cc: *basex-talk@mailman.uni-konstanz.de <
> basex-talk@mailman.uni-konstanz.de>
> *Subject: *Re: [basex-talk] Out of Main Memory
>
> Hi Greg,
>
>
>
> A quick reply: If only parts of your documents are relevant for full-text
> queries, you can restrict the selection with the FTINDEX option (see [1]
> for more information).
>
>
>
> How large is the total size of your input documents?
>
>
>
> Best,
>
> Christian
>
>
>
> [1] https://docs.basex.org/wiki/Indexes#Selective_Indexing
>
>
>
>
>
>
>
> On Tue, Mar 12, 2024 at 8:34 PM Murray, Gregory 
> wrote:
>
> Hello,
>
>
>
> I’m working with a database that has a full-text index. I have found that
> if I iteratively add XML documents, then optimize, add more documents,
> optimize again, and so on, eventually the “optimize” command will fail with
> “Out of Main Memory.” I edited the basex startup script to change the
> memory allocation from -Xmx2g to -Xmx12g. My computer has 16 GB of memory,
> but of course the OS uses up some of it. I have found that if I exit
> memory-hungry programs (web browser, Oxygen), start basex, and then run the
> “optimize” command, I still get “Out of Main Memory.” I’m wondering if
> there are any known workarounds or strategies for this situation. If I
> understand the documentation about indexes correctly, index data is
> periodically written to disk during optimization. Does this mean that
> running optimize again will pick up where the previous attempt left off,
> such that running optimize repeatedly will eventually succeed?
>
>
>
> Thanks,
>
> Greg
>
>
>
>
>
> Gregory Murray
>
> Director of Digital Initiatives
>
> Wright Library
>
> Princeton Theological Seminary
>
>
>
>
>
>

Re: [basex-talk] Trying to get process details on macOS

[Bridger Dyson-Smith]

Ah, apologies ( that's what I get for trying to send emails during bedtime
🙂 )

I have been bitten in the past by arg differences between different Unix
flavors. I have tried the following on Mac OS 12.something with just the
BaseX jar download.

let $ps := proc:execute('ps', ('-o', 'pid,%cpu,command'))
return $ps


PID  %CPU COMMAND
82437   0.0 -bash
82462  16.7 /usr/bin/java -jar BaseX107.jar

  0


This is a borrowed Mac - I didn't install anything. Does your Mac not have
a ps; e.g. `which ps`? This one returns `/bin/ps`.

Sorry for the noise.
Best,
Bridger

On Thu, Oct 19, 2023, 8:00 PM Bridger Dyson-Smith 
wrote:

> Hi Eliot - hope you're well!
>
> Quickly off the cuff, the `ps` command on MacOS is a BSD-flavo(u)r, if I'm
> remembering correctly (it's been a while...).
>
> I don't have a Mac handy right now, but check this `ps` man page -
> https://ss64.com/osx/ps.html - your args may need to be adjusted.
>
> Hope that's helpful.
> Best,
> Bridger
>
> On Thu, Oct 19, 2023, 7:08 PM Eliot Kimber 
> wrote:
>
>> In the context of a 10.6 server running as a web app, I’m trying to
>> implement a function that uses the process module to run commands to get
>> information about the running BaseX servers, in particular, the PID and
>> port each server is on. I start a number of “worker” BaseXHTTP instances to
>> handle long-running processes.
>>
>>
>>
>> On Linux this code works:
>>
>> let $cmdResult := proc:execute(‘ps’, ('-o', 'pid,%cpu,command'))
>>
>>
>>
>> From which I can parse out the PID and port for BaseX processes:
>>
>>
>>
>>   let $lines as xs:string* := ($cmdResult/output/text() =>
>> tokenize('
'))
>>
>>   return
>>
>>   map{
>>
>> 'data' :
>>
>>   let $servers as map(*) := map:merge(
>>
>> for $line at $p in $lines[contains(., 'BaseXHTTP')]
>>
>> let $pid as xs:string := tokenize($line, '\s+')[1]
>>
>> let $port as xs:string := tokenize(substring-after($line, '
>> -p'),'\s+')[1]
>>
>> return map{ $port :
>>
>>   map{
>>
>> 'port' : $port,
>>
>> 'pid'  : $pid
>>
>>   }
>>
>>   }
>>
>>   )
>>
>>   return $servers
>>
>>   }
>>
>>
>>
>> However, on macOS there’s not an exact equivalent of the linux version of
>> ps, but there is the “pgrep” command, which, from the command line, lets me
>> get the data I need:
>>
>> pgrep -lf basex
>>
>>
>>
>> However, when I try to run this from within BaseX, I get “sysmon request
>> failed with error: sysmond service not found pgrep: Cannot get process list”
>>
>>
>>
>> I assume this is a function of how the HTTP servers are running on macOS.
>> I’m starting them like so:
>>
>> "${basexBinDir}/basexhttp" ${debugFlag} -S -p${clientPort} -h${httpPort}
>> -s${stopPort} &
>>
>> So as a service
>>
>> Any ideas how I might make this work on mapOS?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Eliot
>>
>> _
>>
>> *Eliot Kimber*
>>
>> Sr Staff Content Engineer
>>
>> O: 512 554 9368
>>
>> M: 512 554 9368
>>
>> servicenow.com <https://www.servicenow.com>
>>
>> LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter
>> <https://twitter.com/servicenow> | YouTube
>> <https://www.youtube.com/user/servicenowinc> | Facebook
>> <https://www.facebook.com/servicenow>
>>
>

Re: [basex-talk] Trying to get process details on macOS

[Bridger Dyson-Smith]

Hi Eliot - hope you're well!

Quickly off the cuff, the `ps` command on MacOS is a BSD-flavo(u)r, if I'm
remembering correctly (it's been a while...).

I don't have a Mac handy right now, but check this `ps` man page -
https://ss64.com/osx/ps.html - your args may need to be adjusted.

Hope that's helpful.
Best,
Bridger

On Thu, Oct 19, 2023, 7:08 PM Eliot Kimber 
wrote:

> In the context of a 10.6 server running as a web app, I’m trying to
> implement a function that uses the process module to run commands to get
> information about the running BaseX servers, in particular, the PID and
> port each server is on. I start a number of “worker” BaseXHTTP instances to
> handle long-running processes.
>
>
>
> On Linux this code works:
>
> let $cmdResult := proc:execute(‘ps’, ('-o', 'pid,%cpu,command'))
>
>
>
> From which I can parse out the PID and port for BaseX processes:
>
>
>
>   let $lines as xs:string* := ($cmdResult/output/text() =>
> tokenize('
'))
>
>   return
>
>   map{
>
> 'data' :
>
>   let $servers as map(*) := map:merge(
>
> for $line at $p in $lines[contains(., 'BaseXHTTP')]
>
> let $pid as xs:string := tokenize($line, '\s+')[1]
>
> let $port as xs:string := tokenize(substring-after($line, '
> -p'),'\s+')[1]
>
> return map{ $port :
>
>   map{
>
> 'port' : $port,
>
> 'pid'  : $pid
>
>   }
>
>   }
>
>   )
>
>   return $servers
>
>   }
>
>
>
> However, on macOS there’s not an exact equivalent of the linux version of
> ps, but there is the “pgrep” command, which, from the command line, lets me
> get the data I need:
>
> pgrep -lf basex
>
>
>
> However, when I try to run this from within BaseX, I get “sysmon request
> failed with error: sysmond service not found pgrep: Cannot get process list”
>
>
>
> I assume this is a function of how the HTTP servers are running on macOS.
> I’m starting them like so:
>
> "${basexBinDir}/basexhttp" ${debugFlag} -S -p${clientPort} -h${httpPort}
> -s${stopPort} &
>
> So as a service
>
> Any ideas how I might make this work on mapOS?
>
>
>
> Thanks,
>
>
>
> Eliot
>
> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>

Re: [basex-talk] Possible to get line number of node?

[Bridger Dyson-Smith]

Hi Eliot,
Apologies for the mobile guesswork, but what about serializing a temporary
document to look up line number? I'm probably missing something obvious
and/or misunderstanding your challenge, but if I'm not it seems like a
temporary document (in memory? On the filesystem?) might let you get at the
appropriate line numbers.

In any case, good luck!
Best wishes,
Bridger

On Thu, Dec 22, 2022, 11:38 AM Eliot Kimber 
wrote:

> In our Mirabel web application I generate an “Open in Oxygen” button for
> all the maps and topics where appropriate (although we haven’t yet
> implemented named users so it’s not quite ready for use).
>
>
>
> This current challenge is where I’m generating a CSV file as a one-off
> report using BaseX locally and then just providing the resulting
> spreadsheet to the user who requested the report.
>
>
>
> Excel macros is not an option for me.
>
>
>
> I suppose I could just generate the command line value to be pasted into a
> terminal—that should work, if not maximally convenient for the user.
>
>
>
> Cheers,
>
>
>
> E.
>
> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>
>
>
> *From: *Christian Grün 
> *Date: *Thursday, December 22, 2022 at 10:20 AM
> *To: *Zimmel, Daniel 
> *Cc: *Eliot Kimber ,
> basex-talk@mailman.uni-konstanz.de 
> *Subject: *Re: [basex-talk] Possible to get line number of node?
>
> [External Email]
>
> We don’t store any node locations in the database, that’s correct.
> fn:path may be helpful to generate the Oxygen node path.
>
>
> On Thu, Dec 22, 2022 at 11:21 AM Zimmel, Daniel 
> wrote:
> >
> > I can’t answer anything helpful for the reliability of line numbers, but
> can tell you how we are doing this in a similar use case:
> >
> > we make use of what Oxygen calls “simplified XPath index path” (
> https://www.oxygenxml.com/doc/versions/25.0/ug-editor/topics/opening-document-from-cli.html
> )
> >
> >
> >
> > Roughly:
> >
> > * we write out that path (“#element(1/2/6/15)”) to a Schematron
> diagnostics element
> >
> > * in our frontend to Schemaron we generate a button which will take you
> directly to Oxygen
> >
> >
> >
> > So, if you don’t have to care about other editors than Oxygen, this
> might be a simple and reliable way. At least to us, works like a charm.
> >
> >
> >
> > Best, Daniel
> >
> >
> >
> > Von: BaseX-Talk  Im Auftrag
> von Eliot Kimber
> > Gesendet: Mittwoch, 21. Dezember 2022 23:14
> > An: basex-talk@mailman.uni-konstanz.de
> > Betreff: [basex-talk] Possible to get line number of node?
> >
> >
> >
> > I’m pretty sure the answer is “no”, but is there any way from BaseX 9.6
> to get the source line number a given node starts on?
> >
> > My use case is I’m looking for specific elements and making a report of
> documents that contain them and identifying information about the elements
> to help with finding them outside of BaseX (i.e., in Oxygen where the
> authoring happens). I’m parsing with the “trim whitespace” option turned
> off and I’m not doing any DTD-aware parsing, so the elements as loaded into
> the database will match the source (no additional attributes or anything).
> So at least in theory BaseX could maintain accurate line numbers.
> >
> > These elements don’t have @id attributes so no natural way to identify
> them, so it would be useful to be able to report the source line number as
> a guide to people fixing these elements.
> >
> >
> >
> > I didn’t see anything in the docs so I thought I’d ask just in case I’ve
> missed some trick.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > E.
> >
> > _
> >
> > Eliot Kimber
> >
> > Sr Staff Content Engineer
> >
> > O: 512 554 9368
> >
> > M: 512 554 9368
> >
> > servicenow.com
> >
> > LinkedIn | Twitter | YouTube | Facebook
>

Re: [basex-talk] BaseX GraalVM native-image

[Bridger Dyson-Smith]

Hi -

I've had nice luck running either the basexserver + basexclient, or the
basexhttp server (confession, it's almost something I start by default).

Maybe that would be an option to help avoid JVM startup times?
HTH!
Best,
Bridger


On Thu, Dec 1, 2022, 5:08 AM  wrote:

> Hello Everyone,
> Didn't anybydy try and is that possible to convert the BaseX JAR to a
> "native-image" (using GraalVM's convertor or any other tool maybe)?
> The reason: I have to use BaseX in scripts and Makefiles as an XQuery
> engine and have to call it often. The problem is the JVM startup time
> affects the performance significantly when it has to start multiple
> times.

I tried just naïvely running the GraalVM's native-image convertor
> against the BaseX103.jar file and got errors with long backtraces (that
> I didn't quite understand to be honest).
> Did anybody try that, maybe?
> Best regards


>
>
>

Re: [basex-talk] Am I using proc:exec wrong?

[Bridger Dyson-Smith]

Hey Graydon -

I'm on mobile and can't verify, but what about

proc:execute($path-to-java, ('-cp', 'home/graydon/bin/coffeepot-
package/coffeepot-1.99.8/coffeepot-1.99.8.jar', '
org.nineml.coffeepot.Main'))

Multiple args need to be in a sequence, as I recall but I haven't had a
chance to try this with v10.1.

HTH!
Bridger

On Fri, Sep 2, 2022, 6:18 PM Graydon Saunders  wrote:

> Hello --
>
> This is with BaseX 10.1 on a linux box:
> 18:02 graydon % java -version
> openjdk version "17.0.4" 2022-07-19
> OpenJDK Runtime Environment (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8)
> OpenJDK 64-Bit Server VM (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8,
> mixed mode, sharing)
>
> I'm trying to invoke CoffeePot, https://coffeepot.nineml.org/ which is an
> ixml processor.
>
> In the BaseX GUI,
> proc:execute('/usr/bin/java','-version')
>
> gives me the expected version information as an error, despite a return
> code of zero:
> openjdk version "17.0.4" 2022-07-19
> OpenJDK Runtime Environment (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8)
> OpenJDK 64-Bit Server VM (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8,
> mixed mode, sharing)
> 0
>
> From a console,
> /usr/bin/java -cp
> /home/graydon/bin/coffeepot-package/coffeepot-1.99.8/coffeepot-1.99.8.jar
> org.nineml.coffeepot.Main
>
> works fine, in as much as I get the list of coffeepot usage information I
> expect if it's run without arguments.
>
> proc:execute('/usr/bin/java','-cp
> /home/graydon/bin/coffeepot-package/coffeepot-1.99.8/coffeepot-1.99.8.jar
> org.nineml.coffeepot.Main')
>
> returns
> Unrecognized option: -cp
> /home/graydon/bin/coffeepot-package/coffeepot-1.99.8/coffeepot-1.99.8.jar
> org.nineml.coffeepot.Main
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> 1
>
> I suspect I'm doing something wrong, but it's not at all clear to me what.
>
> Anyone have suggestions?
>
> Thanks!
> Graydon
>

Re: [basex-talk] Get All Tokens for Attribute Name?

[Bridger Dyson-Smith]

Hi Eliot -

If I'm following correctly, could you use `index:facets()`? E.g.

index:facets($db)//attribute[@name='bundles']//entry/text() =>
distinct-values()

Can the bundles attribute have multiple tokens?
Best,
Bridger



On Sat, Aug 6, 2022 at 12:01 PM Eliot Kimber 
wrote:

> Using the attribute index I can get all attributes of a specific name that
> have a specific value or token (db:attribute()) and I can get all the
> values for all attributes that start with a specific prefix
> (index:attributes()) but I don’t see a way to get, from the index, all the
> values for all the attributes of a specific name.
>
>
>
> Have I missed something?
>
>
>
> My use case is I have an index over my docs where each index entry is of
> the form:
>
>
>
>filename="c_DataCertification.dita"
> dbpath="product/data-certification/concept/c_DataCertification.dita"
> bundles="bundle-platcap-platform-capabilities"/>
>
>
>
> And I would like the distinct-values of all the @bundles attributes. There
> are about 40K entry elements.
>
>
>
> Of course I can do:
>
> doc-to-bundle-index/doc-to-bundle-index-entry/@bundles ! tokenize(.,
> '\s+')
>
> => distinct-values()
>
>
>
> And that seems plenty fast, but it seemed like there should be a way to do
> it with the indexes alone. But it may be that the query on the @bundle
> attribute actually uses the index anyway…
>
>
>
> Cheers,
>
>
>
> E.
>
>
>
> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>

Re: [basex-talk] Multi-valued values in maps - sequences vs arrays

[Bridger Dyson-Smith]

Andy and Martin -

with apologies, I didn't express my problem very well! Martin, you're
correct with your example

`map { 'key': array { 1 to 5 } }`

and that was part of my struggle; i.e. "It works this way but not that way
- what am I doing wrong?" Andy noticed the source of my dilemma: I a using
map:merge#2, with `map { "duplicates": "combine" }` -- Andy, thank you for
catching that. I was entirely too caught up in a different part of the
documentation.

I'm creating these maps from loose text, where keys are word pairs and
values are value* (0, 1, or many). I'll come back with a better example of
my problem, although I think you've both given me food for thought.
Thank you both so much for your help!
Best,
Bridger




On Wed, Aug 3, 2022 at 6:04 AM Andy Bunce  wrote:

> Hi Bridger,
>
> > is even possible; e.g. map{ "key": [1,2,3,4,5] }
> Well that works fine for me, so yes.
>
> For your other examples, I think the answer is that map:merge with map{
> "duplicates":"combine"} always generates value *sequences *on duplicate
> keys.
> So, maybe, generate the array you want, then put it in the map
>
> let $a:=(1 to 5)!array{.}=>array:join()
> return map:entry("key",$a)
>
> /Andy
>
> On Wed, 3 Aug 2022 at 09:37, Martin Honnen  wrote:
>
>>
>> Am 03.08.2022 um 04:27 schrieb Bridger Dyson-Smith:
>> >
>> >
>> > I would appreciate some help understanding how I might go about having
>> > a multi-valued array as the value of a map key, or if this is even
>> > possible; e.g.
>> >
>> > map{ "key": [1,2,3,4,5] }
>>
>>
>> map { 'key' : array { 1 to 5 } }
>>
>>
>> Or have I missed the point?
>>
>>

[basex-talk] Multi-valued values in maps - sequences vs arrays

[Bridger Dyson-Smith]

Hi all -

I would appreciate some help understanding how I might go about having a
multi-valued array as the value of a map key, or if this is even possible;
e.g.

map{ "key": [1,2,3,4,5] }

I've tried various combinations of array:flatten and array:join, but things
always wind up in a sequence. (Aside: this isn't a real problem, but I'm
trying to force myself to get more comfortable working with arrays and maps
and this seemed like a great exercise... :) My mistake!).

E.g.
map:merge(

  (1 to 5) ! map {'key': array:flatten(array { . }) },
  map{"duplicates":"combine"}
),
(: returns map{"key":(1,2,3,4,5)} :)

map:merge(
  (1 to 5) ! map {'key': array:join(array {.}) },
  map{"duplicates":"combine"}
)

(: returns map{"key":([1],[2],[3],[4],[5])} :)

Are sequences the result because a map is a function item?
Thanks for any clarity you can provide!

Best,
Bridger

Re: [basex-talk] BaseX 10 • The Double-Digit Summer Edition

[Bridger Dyson-Smith]

Christian and company -

As always, thanks for your efforts and work on our behalf!
Congratulations on this significant milestone!

Best,
Bridger

On Mon, Aug 1, 2022 at 9:28 AM Christian Grün 
wrote:

> Dear all,
>
> It’s been around 15 years ago when the first Open-Source version of
> BaseX was released. Thanks to the continuous support of all of you, we
> are thrilled today to announce the first double-digit version of our
> XML framework:
>
>   https://basex.org/
>
> BaseX 10 assists you with your everyday data challenges with some
> great new features:
>
> STORAGE
>
> • Beside XML Documents and Binary Data, you can use the new »XQuery
> Values« resource type to store arbitrary items and sequences in
> databases.
> • General data (registered users, scheduled services, and key-value
> stores) can now be backed up as well.
> • Mixed-content developers will be delighted: Whitespaces are now
> preserved by default.
>
> XQUERY
>
> • Database locking has been improved by splitting up XQuery
> compilation into a static and dynamic step.
> • A new Store Module allows you to organize values in a persistent
> main-memory key-value store.
> • You can store and retrieve XQuery values with db:put-value and
> db:get-value.
> • When serializing XML, indentation is now turned off by default.
>
> GENERAL
>
> • BaseX 10 requires Java 11 to run.
> • The GUI comes with improved HiDPI support, new icons and improved
> consistency.
> • All HTTP requests are now based on the new Java HTTP client.
> • Catalog support is now universal and based on either the new XML
> Catalog API or Norman Walsh’s excellent XML Resolver.
> • The default ports for web applications were changed from 8984/8985
> to 8080/8081.
> • The default admin password was removed to improve security.
> • Results of the REST API are now wrapped in the default REST namespace.
>
> We have taken the version jump as an opportunity to rename various
> existing functions and commands. The database storage has not changed:
> Databases created with BaseX 10 are backward compatible and can still
> be opened with BaseX 9.
>
> Visit our documentation for more details, and information on how to
> upgrade your existing applications:
>
>   https://docs.basex.org/wiki/BaseX_10
>
> Have fun!
> Your BaseX Team
>

Re: [basex-talk] Possible to Speed Up This Lookup?

[Bridger Dyson-Smith]

Hi Eliot -

Others are much more proficient and knowledgeable, but I try the following:

1. ABI := Always Be Indexing :)
2. I think that Christian has expressed that, with indexes in play, many
queries can experience significant optimizations by reversing the query
pattern; eg starting with the attribute value and working back up from
there, etc.
3. The optimized query view can be super helpful in this regard, too.

I'm sure there are other, better habits and strategies, I know (I just
don't have them ingrained! 🙂). Other users will share, I'm sure.

Have a pleasant weekend!
Best,
Bridger

On Fri, Jul 15, 2022, 6:01 PM Eliot Kimber 
wrote:

> I profiled the db:open() and collection() calls and they were basically
> instantaneous.
>
>
>
> Moving to using db:attribute() and the index makes my process 100x times
> faster. Now I feel like a bit of an idiot for not having figured this out
> sooner.
>
>
>
> Somebody needs to write the Optimizing BaseX book….
>
>
>
> Cheers,
>
>
>
> E.
>
>
>
> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>
>
>
> *From: *BaseX-Talk  on behalf
> of Liam R. E. Quin 
> *Date: *Friday, July 15, 2022 at 4:55 PM
> *To: *basex-talk@mailman.uni-konstanz.de <
> basex-talk@mailman.uni-konstanz.de>
> *Subject: *Re: [basex-talk] Possible to Speed Up This Lookup?
>
> [External Email]
>
>
> On Fri, 2022-07-15 at 21:20 +, Eliot Kimber wrote:
> >
> > return (db:option('attrindex'),
> >   for $node in $map
> >   return prof:track(db:open($rkDatabase)/doc-where-used-
> > index/where-used-entry/@key[xs:integer(.) eq 9151416])?time !
> > util:formatTime(.)
>
> can you take the db:open() call out and see how much faster the
> expression runs without it?
>
> return (db:option('attrindex'),
>   let $db := db:open($rkDatabase)
>   return
>   for $node in $map
>
>   return
> prof:track($db/doc-where-used-index/where-used-entry
> /@key[xs:integer(.) eq 9151416])?time ! util:formatTime(.)
>
> (if i have that right)
>
> --
> Liam Quin,
> https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dDcIyNVA$
> 
> Available for XML/Document/Information Architecture/XSLT/
> XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
> Barefoot Web-slave, antique illustrations:
> https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!Ge6_HxIH-uWYEol5ImGje0BVsQ_FKkbp1x9czcNl7d0t6BKvoQ1v0sENn6_kHvOQI0dfFmEig6LEfg3dk_syZfc$
> 
>

Re: [basex-talk] Email integration

[Bridger Dyson-Smith]

Hi Tamara -

On Fri, Jul 8, 2022, 12:07 PM Tamara Marnell 
wrote:

> Hi Ben and Bridger,
>
> I think you'd need to write a separate application or module to handle
> this, or use workarounds. Here are a few options you could explore:
>
> 1) Use the Google API Client libraries
> <https://developers.google.com/api-client-library> to write an
> application in your preferred language that queries all messages and saves
> them in XML to use in a BaseX DB. I've used the PHP client library for
> Gmail, Drive, Groups, and Analytics reports, and it made the initial OAuth
> token setup relatively painless (as in, I was only tearing my hair out for
> a single day, not multiple).
>

This was the intent of my earlier email - a module that wraps the Java
client(s) - but your thoughts are much more thorough! I appreciate the
suggestions for alternative langs, too. Good to know about the limited
potential for self-driven hair loss, too - it's a concern :) :).

>
> 2) Write a script in your preferred language that uses IMAP to fetch and
> save the messages. PHP <https://www.php.net/manual/en/book.imap.php>, Java
> <https://javaee.github.io/javamail/>, and Python
> <https://docs.python.org/3/library/imaplib.html> have built-in or
> third-party client libraries. But as Ben said, authentication will be a
> pain because Gmail won't allow apps to authenticate with just usernames and
> passwords anymore. You'd need to build in OAuth with your own methods.
>
> 3) Find an existing web client to sync to your Gmail, and find where it
> stores local copies of the messages. Use them to create a BaseX DB. It
> might be difficult or impossible to find one that saves messages in neat,
> convenient XML files, though, so you might have to export an archive and do
> some file processing processing every time you want to perform a query.
>
> -Tamara
>


Thanks for the email!
Best,
Bridger



>
>
> On Tue, Jul 5, 2022 at 7:25 PM Bridger Dyson-Smith 
> wrote:
>
>> Hi Ben -
>>
>>
>> On Tue, Jul 5, 2022 at 7:36 PM Ben Pracht  wrote:
>>
>>> Hi,
>>> This may seem a bit unusual, but I like the parsing capabilities of
>>> XQuery and I'd like to use it for my email. I'd like to have BaseX log into
>>> my IMAP server, fetch emails and put each email as a separate XML document.
>>> I'm willing to have it ignore attachments if necessary. Then, I'd like to
>>> query my email using normal BaseX XQuery. This isn't for normal emails, but
>>> I wanted to be able to query it for automated emails and for one-off type
>>> queries.
>>>
>>> Oh, to make things more difficult, I'm using GMail, so the login just
>>> got even more difficult.
>>>
>> Any suggestions?
>>>
>>> An(other!) idea I've been kicking around (and wondering about) but not
>> testing/exploring/trying: integrating the Google OAuth2 api client(s)
>> [1,2]. I don't know how it would work (or fit together), but that seems
>> like a reasonable place to start.
>>
>>
>> Thanks,
>>> Ben
>>>
>>
>> I hope this is helpful.
>> Best,
>> Bridger
>>
>> [1]
>> https://developers.google.com/api-client-library/java/google-api-java-client/oauth2
>> [2] https://github.com/googleapis/google-auth-library-java
>>
>>
>>
>>> Best Regards,
>>> Ben Pracht
>>> 919.809.2439
>>> ben.pra...@gmail.com
>>>
>>
>
> --
>
> Tamara Marnell
> Program Manager, Systems
> Orbis Cascade Alliance (orbiscascade.org <https://www.orbiscascade.org/>)
> Pronouns: she/her/hers
>

Re: [basex-talk] Email integration

[Bridger Dyson-Smith]

Hi Ben -


On Tue, Jul 5, 2022 at 7:36 PM Ben Pracht  wrote:

> Hi,
> This may seem a bit unusual, but I like the parsing capabilities of XQuery
> and I'd like to use it for my email. I'd like to have BaseX log into my
> IMAP server, fetch emails and put each email as a separate XML document.
> I'm willing to have it ignore attachments if necessary. Then, I'd like to
> query my email using normal BaseX XQuery. This isn't for normal emails, but
> I wanted to be able to query it for automated emails and for one-off type
> queries.
>
> Oh, to make things more difficult, I'm using GMail, so the login just got
> even more difficult.
>
Any suggestions?
>
> An(other!) idea I've been kicking around (and wondering about) but not
testing/exploring/trying: integrating the Google OAuth2 api client(s)
[1,2]. I don't know how it would work (or fit together), but that seems
like a reasonable place to start.


Thanks,
> Ben
>

I hope this is helpful.
Best,
Bridger

[1]
https://developers.google.com/api-client-library/java/google-api-java-client/oauth2
[2] https://github.com/googleapis/google-auth-library-java



> Best Regards,
> Ben Pracht
> 919.809.2439
> ben.pra...@gmail.com
>

Re: [basex-talk] BaseX 9.7.3: Tweaks, Fixes, Previews

[Bridger Dyson-Smith]

Christian and team -

On Thu, Jun 30, 2022 at 10:04 AM Christian Grün 
wrote:

> Hi all,
>
> A new BaseX maintenance release is available. This is what you get:
>
> • DBA: Improved support for database backups
> • Performance tweaks and bug fixes
>
> …and some more BaseX 10 previews:
>
> • Store Module (before: Caching Module)
> • Backup of general data (users, jobs, stores)
>
> We spend most of our time on BaseX 10. Stay tuned!
>
>
Thank you for your efforts on our behalf!
I won't speak for others, but I'm on the edge of my seat! Looking forward
to v10.

Yours,
> Christian
> BaseX Team
>

Best wishes,
Bridger

Re: [basex-talk] BaseX 9.7.3: Tweaks, Fixes, Previews

[Bridger Dyson-Smith]

On Thu, Jun 30, 2022 at 10:04 AM Christian Grün 
wrote:

> Hi all,
>
> A new BaseX maintenance release is available. This is what you get:
>
> • DBA: Improved support for database backups
> • Performance tweaks and bug fixes
>
> …and some more BaseX 10 previews:
>
> • Store Module (before: Caching Module)
> • Backup of general data (users, jobs, stores)
>
> We spend most of our time on BaseX 10. Stay tuned!
>
> Yours,
> Christian
> BaseX Team
>

Re: [basex-talk] Link Graph Construction: Anything I Can Crib or Learn From?

[Bridger Dyson-Smith]

Hi Eliot -

I've wondered (but never tested/explored) about leveraging some semblance
of json-ld (or serialized ttl, or something similar) and passing those
values to Apache Jena (or another SPARQL processor) to use that as an
inference engine. I'm deep in Speculation Territory here - I don't know
what anything would look like - but you're describing an interesting
problem, and it seems doable. Martynas Jusevicius (and his colleagues) have
a project, LinkedDataHub, that may provide another avenue for exploring
this -- I haven't used AtomGraph's applications, but he's active on the
xml.com slack, and it looks like there are some interesting visualization
capabilities with their work.

Our listserv friend and neighbor, Tim Thompson of Yale, may have some ideas
along these lines, too.
Sorry that I can't provide anything concrete, but I hope some of this is
somewhat helpful.
Best,
Bridger

[1] https://json-ld.org/
[2] https://jena.apache.org/
[3] https://github.com/AtomGraph/LinkedDataHub

On Thu, Jun 23, 2022 at 10:35 AM Eliot Kimber 
wrote:

> In the context of our Project Mirabel system that manages DITA content, I
> need to be able answer the question “for topic X, what other topics link to
> it directly or indirectly?”
>
>
>
> That is, say Topic A links to Topic B that Links to Topic C.
>
>
>
> Asking the question “What topics ultimately link to topic C?” I would like
> to get the answer “Topic A, Topic B”.
>
>
>
> Getting the answer for direct references is easy—I already build a
> where-used index that captures, for each DITA map or topic, what other maps
> and topics link directly to it.
>
>
>
> But to get the Topic A part of the answer I need some kind of link graph
> index and I’m not sure how best to go about calculating this or capturing
> it in some index or set of indexes.
>
>
>
> In our content the fan out from a single Topic C to the set of topics that
> ultimately reference it could be 10s of 1000s of topics. We have about 45K
> topics in the content for each version of the ServiceNow Platform and a
> number of topics that are used by a large number of other topics, so the
> explosion can be quite large. That suggests that a simple
> topic-to-ultimately-referenced-topics index would be very inefficient in
> that the entry for any given topic could potentially have 45K – 1 entries
> (we don’t care that a topic references itself).
>
>
>
> On the other hand, working backwards through chains of direct references
> can also be expensive and is probably too slow, so maybe the brute-force
> index is the best option?
>
>
>
> At the same time, I would like to be able to quickly visualize the link
> graph extending from or ending in any given topic or simply the link graph
> for the entire information set, which requires capturing the nodes and
> edges.
>
>
>
> My question: does anyone either have experience or insight into this kind
> of link graph challenge or know of relevant papers or general discussion of
> graph processing I might look at?
>
>
>
> Thanks,
>
>
>
> Eliot
>
> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>

Re: [basex-talk] Encoding hassle ....

[Bridger Dyson-Smith]

Marco -
I'm sorry but I can only corroborate your findings, and that trying to
force UTF-8 by adding the encoding parameter to the functions doesn't seem
to help; e.g.

) ./bin/basex
BaseX 9.7.1 [Standalone]
Try 'help' to get more information.
> xquery file:current-dir()
/usr/home/bridger/bin/basex/
Query executed in 886.62 ms.
> xquery file:write-text("a1.txt", "°" || out:nl(), "UTF-8")

Query executed in 4.32 ms.
> xquery file:read-text("a1.txt")
°

Query executed in 1.99 ms.
> xquery file:write-text("a2.txt", file:read-text("a1.txt", "UTF-8"),
"UTF-8")

Query executed in 1.83 ms.
> xquery file:read-text("a2.txt")
[file:io-error] Decoding error: xb0
> xquery file:read-text("a2.txt", "UTF-8")
[file:io-error] Decoding error: xb0
> xquery file:read-text("a2.txt", "ISO-8859-1")
°

Query executed in 2.01 ms.

On Fri, May 27, 2022 at 1:00 PM Marco Lettere  wrote:

> Dear all,
>
> after wrapping our heads around this for hours today, we don't know how
> to get rid of this inconsistency. Thus I ask for help ...
>
> SSCE:
>
> BaseX 9.6.4 [Standalone]
> Try 'help' to get more information.
>  > xquery file:write-text("a1.txt", "°" || out:nl()) (: Same with
> codepoints-to-string(176) instead of "°" :)
>
> Query executed in 183.94 ms.
>  > xquery file:read-text("a1.txt")
> °
>
> Query executed in 1.49 ms.
>  > xquery file:write-text("a2.txt", file:read-text("a1.txt"))
> Query executed in 3.4 ms.
>
>  > xquery file:read-text("a2.txt")
> [file:io-error] Decoding error: xb0
>
> Testing the files with linux command-line tool "file", this is the output:
>
>  > file a1.txt
> a1.txt: Unicode text, UTF-8 text
>
>  > file a2.txt
> a2.txt: ISO-8859 text
>
> Reading the file after "copying" it seems to change the encoding. How is
> this supposed to be handled?
>
> Regards,
>
> Marco.
>
>

Re: [basex-talk] Selecting nodes in a db by source file name

[Bridger Dyson-Smith]

Charles -

On Thu, May 19, 2022 at 12:46 PM Charles Bearden  wrote:

> Thanks to Graydon, Tamara, and Christian for responding!
>
> I figured out a pretty fast way to exploit the infrastructure I had built
> (the files allocated out into many databases and a single index database
> generated from the databases).
>
> Here is a sample record from my index database:
>
>
>
>
>
>
>
> *  pmed_updates_b  34239076
> 1  pubmed22n1145.xml
> 2022-01-09*
>
> As it happens, there are eight versions of this record scattered across 7
> of the component databases and located in 8 input files (two of the input
> files were allocated to one of the databases). Each of these instances has
> an entry in the index database.
>
> My approach has four steps:
>
>1. retrieve all entries from the index database that have the desired
>PMID;
>2. convert the sequence of XML entries into a sequence of maps with
>the same data, ordering by filename descending, so that the most recent
>file is the first element of the sequence;
>3. take the first item/map of the sequence;
>4. look up all occurrences of records with that PMID in the database
>specified in the first item and call *db:path()* on each item and
>compare it to the filename specified in the most recent record; the record
>whose *db:path()* matches the item/map taken in step three is the most
>recent version of the record with that PMID.
>
> Files are allocated by modulo to the different databases, so it is
> conceivable that a database will have more than one record with a given
> PMID, hence the necessity of comparing each record's path with the one
> given in the map from step three to determine which is the most recent.
>
> Very neat. I had a thought that `db:list-details()`, specifically the 2nd
signature, would be useful here but now that I've 1) read your solution,
and 2) tried to play with some examples, I don't think it would be a very
helpful fit.


> Given the above PMID (for which there are eight versions of the record, as
> noted above) it took less than half a second to retrieve the most recent
> instance of that record out of over 35 million records.
>
> I can post the XQuery if anyone wants to see it. It would take longer to
> document how I build the content & index databases, and I still have to
> work out the best way to keep it all up to date.
>
> Selfishly, I'd be very interested in seeing examples but don't put
yourself through any trouble.

All the best,
> Chuck
>
--
> Sr Systems Analyst
> University of Texas M.D. Anderson Cancer Center
>
>

Thanks for the interesting example.
Best,
Bridger

Re: [basex-talk] Stemming in BaseX Full-Text

[Bridger Dyson-Smith]

Tim -

On Wed, Apr 13, 2022, 4:53 PM Tim Thompson  wrote:

> Thanks, Bridger--that's very helpful! I'm not sure what MarkLogic is using
> exactly, but it seems fairly sophisticated (there's even an advanced option
> for multiple stemming: e.g., "further" has "far," "farther," "further" as
> stems).
>

Indeed, MarkLogic appears to offer a number of stemmers, some of which
offer lemmatization functionality. I couldn't say if adding this type of
capability to BaseX is feasible.

>
> All best,
> Tim
>
>
Best,
Bridger


> --
> Tim A. Thompson (he, him)
> Librarian for Applied Metadata Research
> Yale University Library
>
>
>
> On Wed, Apr 13, 2022 at 12:13 PM Bridger Dyson-Smith <
> bdysonsm...@gmail.com> wrote:
>
>> Hi Tim  -
>>
>> On Wed, Apr 13, 2022 at 11:40 AM Tim Thompson  wrote:
>>
>>> I'm currently involved in a project that's using MarkLogic, and I
>>> noticed that its implementation of English-language stemming differs from
>>> that of BaseX: e.g., "mouse" and "mice" both stem to "mouse."
>>>
>>> In BaseX, those words are stemmed separately. Is this a known limitation
>>> of the internal English syntax parser?
>>>
>>> It's my (admittedly, *VERY*) limited understanding that the BaseX
>> stemmer, at least for English, is limited to the Porter Stemmer[1]. The
>> Porter Stemmer just stems, and doesn't handle stemming from plurals to
>> singulars in the case of apophonic plurals.
>>
>> It'd be interesting to learn what stemmer(s) MarkLogic uses.
>>
>> And, while I'm not that familiar with it (and it would probably entail
>> significant work to implement), the `ft:thesaurus()` function provides
>> similar functionality:
>> ```
>> ft:thesaurus(
>>   
>> 
>>   mice
>>   
>> mouse
>> NT
>>   
>>   
>> rodent
>> BTG
>>   
>> 
>>   ,
>>   'mice'
>> )
>> ```
>>
>>
>>> Example:
>>>
>>> db:create("stem-test",
>>>   
>>> mouse
>>> mice
>>>   
>>>   , "data", map {"ftindex": true(), "stemming": true(), "language": "en"}
>>> )
>>> ,
>>> update:output(
>>>   ft:search("stem-test", "mice")
>>> )
>>>
>>>
>>> Thanks,
>>> Tim
>>>
>>>
>>>
>> Best,
>> Bridger
>>
>> [1]
>> https://github.com/BaseXdb/basex/blob/da1e55d0214e44c1532f121c282021db50a9aa51/basex-core/src/main/java/org/basex/util/ft/EnglishStemmer.java
>>
>>
>> --
>>> Tim A. Thompson (he, him)
>>> Librarian for Applied Metadata Research
>>> Yale University Library
>>>
>>>

Re: [basex-talk] Stemming in BaseX Full-Text

[Bridger Dyson-Smith]

Hi Tim  -

On Wed, Apr 13, 2022 at 11:40 AM Tim Thompson  wrote:

> I'm currently involved in a project that's using MarkLogic, and I noticed
> that its implementation of English-language stemming differs from that of
> BaseX: e.g., "mouse" and "mice" both stem to "mouse."
>
> In BaseX, those words are stemmed separately. Is this a known limitation
> of the internal English syntax parser?
>
> It's my (admittedly, *VERY*) limited understanding that the BaseX stemmer,
at least for English, is limited to the Porter Stemmer[1]. The Porter
Stemmer just stems, and doesn't handle stemming from plurals to singulars
in the case of apophonic plurals.

It'd be interesting to learn what stemmer(s) MarkLogic uses.

And, while I'm not that familiar with it (and it would probably entail
significant work to implement), the `ft:thesaurus()` function provides
similar functionality:
```
ft:thesaurus(
  

  mice
  
mouse
NT
  
  
rodent
BTG
  

  ,
  'mice'
)
```


> Example:
>
> db:create("stem-test",
>   
> mouse
> mice
>   
>   , "data", map {"ftindex": true(), "stemming": true(), "language": "en"}
> )
> ,
> update:output(
>   ft:search("stem-test", "mice")
> )
>
>
> Thanks,
> Tim
>
>
>
Best,
Bridger

[1]
https://github.com/BaseXdb/basex/blob/da1e55d0214e44c1532f121c282021db50a9aa51/basex-core/src/main/java/org/basex/util/ft/EnglishStemmer.java


-- 
> Tim A. Thompson (he, him)
> Librarian for Applied Metadata Research
> Yale University Library
>
>

Re: [basex-talk] Enclosing strings by double quotes for CSV data output

[Bridger Dyson-Smith]

Markus -

Another confession: I've never been very good at puzzles and riddles :(
so...

On Wed, Apr 6, 2022 at 10:34 AM Markus Elfring 
wrote:

> > > Test result:
> > > T3|T4
> > > "line 1
> > > line 2?"|line 3\nline 4?
> > >
> > >
> > > Will any more clarification help here?
> >
> > …, but if you're asking "Why are there double quotes around my `T3`
> field?",
>
> Partly, yes.
> (But not really.)
>
> That's certainly a cryptic response! :)


>
> > then the answer appears to be in the CSV specification[1].
>
> Which impressions did you get from the other BaseX information sources
> which I pointed out?
>
> My impressions with the BaseX documentation, generally speaking, are very
positive, and when I have questions (or need clarification, or help, etc),
my impressions of the community here are even more positive: what a
welcoming and generous group of people. I'm lucky to participate.


> > So, the behavior you're seeing in your sample script matches the CSV RFC.
>
> What is your opinion for the data processing of the fields “T1”, “T2” and
> “T4”?
>

Wrapping up the other examples you've shared in this mail thread, along
with some changes:
e.g.
```
xquery version "3.1";
declare option output:method "csv";
declare option output:csv "header=yes,quotes=yes,separator=|";
declare variable $input :=

  
01
X Y
Y, X
line 4
or a line break in the T3 field.
line 6\nor something - maybe this is line 7? It might not be: what
interprets the '\n' character? BaseX? Something else? Maybe we want the {
string-join(('&','#','10',';','')) } character (linefeed literal)?
  
  
02
A 2nd row
  
;


  {
for $row in $input//row
return (
  
{
  for $field in $row/child::*
  return (
$field
  )
}
  
)
  }

```

Provides the results that I would expect:
```
ID|T1|T2|T3|T4
01|X Y|Y, X|"line 4
or a line break in the T3 field."|line 6\nor something - maybe this is
line 7? It might not be: what interprets the '\n' character? BaseX?
Something else? Maybe we want the 
 character (linefeed literal)?
02|A 2nd row|||
```
My opinion of these results is that they are correct - they meet my
expectations. Do you feel like these results are missing something? If so,
what specifically?
Is there something in the documentation that could change to make things
more explicit or provide better clarity?


>
> Regards,
> Markus
>

Best,
Bridger

Re: [basex-talk] Enclosing strings by double quotes for CSV data output

[Bridger Dyson-Smith]

Hi Markus -

On Wed, Apr 6, 2022 at 9:25 AM Markus Elfring  wrote:

> > No quotes will be added, as your delimiter does not occur in the text
> value.
>
> How does this feedback fit to previously provided information?
>
> A)
> https://docs.basex.org/wiki/CSV_Module#Options
> “…
> Serialization: If the option is enabled, the value will be wrapped with
> quotes if it contains characters that might be treated as control
> characters.
> …”
>
> B)
> https://mailman.uni-konstanz.de/pipermail/basex-talk/2022-April/017038.html
> “…
> The double quote is the text qualifier that’s used for serializing fields
> with spaces.
> …”
>
>
> Another XQuery script example:
> declare option output:method "csv";
> declare option output:csv "header=yes, quotes=yes, separator=|";
> 
> 
> line 1
> line 2?
> line 3\nline 4?
> 
> 
>
> Test result:
> T3|T4
> "line 1
> line 2?"|line 3\nline 4?
>
>
> Will any more clarification help here?
>
> I confess that I'm not sure, but if you're asking "Why are there double
quotes around my `T3` field?", then the answer appears to be in the CSV
specification[1]. So, the behavior you're seeing in your sample script
matches the CSV RFC.

If that wasn't your question, it might help to restate, or reframe, your
questions and concerns.


> Regards,
> Markus
>
Best wishes,
Bridger

[1] https://datatracker.ietf.org/doc/html/rfc4180#section-2, specifically
part 6:

Fields containing line breaks (CRLF), double quotes, and commas
>should be enclosed in double-quotes.  For example:
>
>"aaa","b CRLF
>bb","ccc" CRLF
>zzz,yyy,xxx
>
>

Re: [basex-talk] copy-paste, can't copy in BaseX and then paste into Emacs (Debian)

[Bridger Dyson-Smith]

Patrick -
Glad that helped. I find myself working with Unix-like OS most of the time,
and I'm very accustomed to the XWindows "use the middle mouse button to
copy highlighted selection" behavior. I've noticed that some applications
don't support this behavior - BaseX GUI is one of them - and I think the
commonality is Java... Swing (? at least as I type this I think that Swing
is the commonality).

Best,
Bridger
PS Does MS Windows support this kind of thing? I'm not sure - I don't have
a Windows machine handy

On Fri, Apr 1, 2022 at 11:51 AM Patrick Durusau  wrote:

> Bridger,
>
> You're right!
>
> I has copy setup on highlighting and that doesn't seem to work, but your
> suggestion does.
>
> Thanks!
>
> Patrick
>
> On 4/1/22 11:22, Bridger Dyson-Smith wrote:
> > Hi Patrick -
> >
> > I was able to copy (ctrl-c) out of the Result window and paste into
> > Emacs with ctrl-y.
> > Is that not working for you?
> >
> > Best,
> > Bridger
> >
> > On Fri, Apr 1, 2022 at 10:34 AM Patrick Durusau 
> > wrote:
> >
> > Hello,
> >
> > I tried to copy a string out of results with the copy command on the
> > drop down but don't seem to be able to paste into Emacs.
> >
> > Trying to be good and document my steps but if I can't copy-paste,
> > that's going to be tedious.
> >
> > Suggestions?
> >
> > Thanks!
> >
> > Patrick
> >
> > --
> > Patrick Durusau
> > patr...@durusau.net
> > Technical Advisory Board, OASIS (TAB)
> > Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300
> > Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps)
> >
> > Another Word For It (blog): http://tm.durusau.net
> > Homepage: http://www.durusau.net
> > Twitter: patrickDurusau
> >
> --
> Patrick Durusau
> patr...@durusau.net
> Technical Advisory Board, OASIS (TAB)
> Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300
> Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps)
>
> Another Word For It (blog): http://tm.durusau.net
> Homepage: http://www.durusau.net
> Twitter: patrickDurusau
>
>

Re: [basex-talk] copy-paste, can't copy in BaseX and then paste into Emacs (Debian)

[Bridger Dyson-Smith]

Hi Patrick -

I was able to copy (ctrl-c) out of the Result window and paste into Emacs
with ctrl-y.
Is that not working for you?

Best,
Bridger

On Fri, Apr 1, 2022 at 10:34 AM Patrick Durusau  wrote:

> Hello,
>
> I tried to copy a string out of results with the copy command on the
> drop down but don't seem to be able to paste into Emacs.
>
> Trying to be good and document my steps but if I can't copy-paste,
> that's going to be tedious.
>
> Suggestions?
>
> Thanks!
>
> Patrick
>
> --
> Patrick Durusau
> patr...@durusau.net
> Technical Advisory Board, OASIS (TAB)
> Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300
> Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps)
>
> Another Word For It (blog): http://tm.durusau.net
> Homepage: http://www.durusau.net
> Twitter: patrickDurusau
>
>

Re: [basex-talk] BaseX 9.7 • Ukraine Edition

[Bridger Dyson-Smith]

Dear Christian, and the rest of the BaseX team -

many thanks for your thoughtful message. I'll echo Marco's comments, and
say I'm excited to see where we go in the upcoming decades!

Best wishes,
Bridger

On Wed, Mar 23, 2022 at 8:16 AM Christian Grün 
wrote:

> Dear all,
>
> Just next to us, millions of people are currently being torn out of
> their home country, and we desperately hope that the situation in
> Ukraine will calm down soon. For those of you who are planning to
> donate money to us in the upcoming weeks, we kindly ask you this time
> to redirect it to Médecins Sans Frontières or any NGO of your choice.
>
> – We are back to provide you with a new and fresh version of BaseX. It
> comes with a big bunch of advanced performance tweaks and
> optimizations and some new XQuery features:
>
> • Support for xsl:messages via xslt:transform-report:
> • Support for formatting integers and dates in French
> • XSD validation: Improved support for XML catalogs
> • Remove namespaces from elements via util:strip-namespaces
>
> For a more comprehensive list of added and updated features, look into
> our documentation (docs.basex.org) and check out the GitHub issues
> (github.com/BaseXdb/basex/issues).
>
> And one more pleasant announcement: Today, our company BaseX GmbH Our
> celebrates its 10th anniversary! Thanks to all of you who have keeping
> us busy with challenging projects. Rest assured we’ll be happy to
> continue to give you advice and support for the upcoming decades.
>
> All the best,
> Your BaseX Team
>

Re: [basex-talk] OpenJDK vs. Oracle JDK versus ...

[Bridger Dyson-Smith]

Hi Jonathan -

I am by no means any kind of power user when it comes to JDKs, but if
you'll entertain an anecdote:
I have read in many places that unless your Java is written specifically to
take advantage of Oracle's JVM, then you're
fine to use OpenJDK (or Azul, or Amazon's offering, or ___).

But others will almost certainly know better :)

HTH
Best,
Bridger

On Tue, Mar 22, 2022 at 3:22 PM Jonathan Robie 
wrote:

> I am setting up a Windows 11 box.
>
> Which Java is best for BaseX?
>
> Jonathan
>

Re: [basex-talk] Dumb question: Adding files to BaseX database

[Bridger Dyson-Smith]

Hi Patrick -

You'll want to navigate to the Database menu > Properties > and you should
find an Add Files dialog there.

Hope that's helpful.
Best,
Bridger

On Mon, Mar 14, 2022 at 1:45 PM Patrick Durusau  wrote:

> Greetings!
>
> I'm using db:add to add files to a BaseX database, but am I just missing
> how to do that in the GUI?
>
> Thanks!
>
> Patrick
>
> --
> Patrick Durusau
> patr...@durusau.net
> Technical Advisory Board, OASIS (TAB)
> Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300
> Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps)
>
> Another Word For It (blog): http://tm.durusau.net
> Homepage: http://www.durusau.net
> Twitter: patrickDurusau
>
>

Re: [basex-talk] Develop a module, HOWTO

[Bridger Dyson-Smith]

Ben,

Apologies - I think I misread your email (and let some things I was
thinking of steer the linking!).

For an XQuery module; e.g.
utilities.xqm
```
xquery version "3.1";
module namespace utils = "http://canofbees.org/ns/xquery/utilities";;

declare function utils:atty(
  $att as attribute()
) as xs:string* {
  "[@" || name($att) || "='" || data($att) || "']"
};

declare function utils:pathing(
  $result as xs:string,
  $node as node()
) {
  concat(
$result, "/"[$result], name($node),
$node/@* ! utils:atty(.) => string-join("")
  )
};

declare function utils:build-path(
  $nodes as node()*
) as xs:string* {
  distinct-values(
$nodes//* ! fold-left(ancestor-or-self::*, '', utils:pathing(?, ?))
  )
};
```

and is called from an XQuery by:
```
xquery version "3.1";
import module namespace utils = "http://canofbees.org/ns/xquery/utilities";;
declare variable $input :=
  
one

  ;

utils:build-path($input)
```

I have this particular module installed in my $BASEX/repo directory - more
on that in the packaging link from earlier.
Hope that's more helpful :) (and closer to the mark!)

Best,
Bridger

On Fri, Feb 25, 2022 at 8:35 AM Bridger Dyson-Smith 
wrote:

> Hi Ben,
>
>
> On Fri, Feb 25, 2022 at 8:25 AM Ben Engbers 
> wrote:
>
>> Hi,
>>
>> I know that it is possible to create a module with functions (I have
>> even done that once), but I can't find the documentation anymore on how
>> to do that.
>> Could please someone provide the URL to this information?
>>
>> Here is the specification: http://expath.org/spec/pkg
> and the BaseX-specific documentation:
> https://docs.basex.org/wiki/Repository
>
> If those aren't the pages you're thinking of, please say! :)
>
> Thanks,
>>
>
> Ben Engbers
>>
>
> HTH
>
> best,
> Bridger
>

Re: [basex-talk] Develop a module, HOWTO

[Bridger Dyson-Smith]

Hi Ben,


On Fri, Feb 25, 2022 at 8:25 AM Ben Engbers 
wrote:

> Hi,
>
> I know that it is possible to create a module with functions (I have
> even done that once), but I can't find the documentation anymore on how
> to do that.
> Could please someone provide the URL to this information?
>
> Here is the specification: http://expath.org/spec/pkg
and the BaseX-specific documentation: https://docs.basex.org/wiki/Repository

If those aren't the pages you're thinking of, please say! :)

Thanks,
>

Ben Engbers
>

HTH

best,
Bridger

Re: [basex-talk] string-join with a newline separator?

[Bridger Dyson-Smith]

Hi Ben,

On Thu, Feb 24, 2022 at 10:44 AM Ben Engbers 
wrote:

> Hi,
>
> My xml has the structure
> 
>
>  
>bla
>  
>
>
>  
>bla
>  
>  
>bla
>  
>
> 
>
> The  element contains 1 to many  elements.
>
> let $tekst := fn:string-join(fn:data($par//al/text()), ".") concatenates
> this to:
> bla.bla.bla
> But I want it to return:
> bla
> bla
> bla
>
> Is it possible to add a newline item-separator to fn:string-join?
>
> how about string-join($input//al/text(), '
')?



> Ben Engbers
>
Best,
Bridger

Re: [basex-talk] feature request: opening database at arbitrary file path

[Bridger Dyson-Smith]

Hi Eric,

On Mon, Feb 21, 2022, 11:44 PM Eric Levy  wrote:

> On Mon, 2022-02-21 at 23:33 -0500, Liam R. E. Quin wrote:
> > On Mon, 2022-02-21 at 20:00 -0500, Eric Levy wrote:
> > > My use would need a function as such that may be used as such:
> > >
> > > db:open_from_path("/home/user/path/to/database/in/filesystem")
> >
> > For that you would want to set DBPATH in a separate BaseX instance, i
> > think. Once a BaseX server is running it doesn't want to switch
> > database directory.
> >
> > You can have multiple databases in the same folder (identified by
> > DBPATH) though. Each database is contained in its own subdirectory.
> >
> > The standalone basex can be given a DBPATH option too. Or you could
> > use
> > symbolic links to map from database name to other directories.
>
> Yes, I think the suggestion was given previously for using DBPATH. I
> need to consider how easy it is to accept. It is a workaround, and
> gives me some hesitation about robustness and stability.
>
> It seems also that BaseX writes files under the user home directory,
> which unfortunately is not an agreeable side effect for my application.
>

BaseX would write files to the specified directory, based on configuration.
Perhaps the docs would help clarify [1].

HTH!
Best,
Bridger

[1] https://docs.basex.org/wiki/Configuration#Home_Directory

>
>

Re: [basex-talk] Long-running queries: command line or BaseX Gui?

[Bridger Dyson-Smith]

Hi Jonathan

On Wed, Feb 16, 2022, 7:04 PM Jonathan Robie 
wrote:

> Do long-running queries run faster from the command line, the GUI, or
> about the same from either?
>

Without a recent example to pull from, I want to say about the same amount
of time (but, obviously, with the preceding caveat).



> Jonathan
>
Hopefully that's a help.
Best wishes,
Bridger

>
>

Re: [basex-talk] How to extract value from fn:analyze-string

[Bridger Dyson-Smith]

Gerrit!

You wound me - I'm cut to the very core :)

For the life of me, I couldn't remember if the `fn` prefix was correct or
not.

@Ben - Gerrit's absolutely right - wildcards are desperate measures (or for
quick, one-off scripts). Using the namespace is much preferable.

Best,
Bridger

On Thu, Feb 10, 2022, 12:46 PM Imsieke, Gerrit, le-tex <
gerrit.imsi...@le-tex.de> wrote:

> It’s a namespace thing. The analyze-string() result is in the
> http://www.w3.org/2005/xpath-functions namespace, which is bound to the
> fn prefix. So you should write fn:match etc. instead of match, or, as
> Bridger suggested, *:match. But such a wildcard always seems a bit
> desperate to me (no offense, Bridger ;).
> Whereas you don’t need to use the privileged fn prefix when you invoke
> analyze-string(), it’s only important when you select the namespaced
> results.
>
> Gerrit
>
> On 10.02.2022 18:30, Ben Engbers wrote:
> > Hi,
> >
> > This query produces the following result:
> >
> > let $debates := collection("Parliament")
> > for $debate-item in $debates
> >let $item-file := $debate-item/officiele-publicatie//meta/@content
> >let $debate-id := fn:analyze-string(
> >  $debate-item/officiele-publicatie//meta/@content,
> "(\d{8}-\d*)-(\d*)")
> >return ($debate-id)
> >
> > =>
> >
> > http://www.w3.org/2005/xpath-functions";>
> >https://zoek.officielebekendmakingen.nl/h-tk-
> >
> >  20202021-102-1
> >
> >/metadata.xml
> > 
> > ...
> >
> > I am trying to extract the values from group 1 and 2 but this query
> > returns 0 results:
> >
> > let $debates := collection("Parliament")
> > for $debate-item in $debates
> >let $item-file := $debate-item/officiele-publicatie//meta/@content
> >let $debate-id := fn:analyze-string(
> >  $debate-item/officiele-publicatie//meta/@content,
> "(\d{8}-\d*)-(\d*)")
> >
> >  let $debate-nr := $debate-id//match/group[@nr="1"]/text()
> >  let $item-nr := $debate-id//match/group[@nr="2"]/text()
> >
> >return ($debate-nr, $item-nr)
> >
> > My guess is that analyze-string inserts new elements in the query and
> > that that is the reason why this does not work.
> >
> > How can I extract debate-nr and item-nr from $debate-id?
> >
> > Ben Engbers
>
>

Re: [basex-talk] How to extract value from fn:analyze-string

[Bridger Dyson-Smith]

Include the namespace, or put in a wildcard; eg

let $debate-nr := $debate-id//*:match/*:group[@nr="1"]/text()
 let $item-nr := $debate-id//*:match/*:group[@nr="2"]/text()

I'm mobile at the moment, but think that's right (: untested and off the
cuff, in other words :).

Best,
Bridger


On Thu, Feb 10, 2022, 12:30 PM Ben Engbers 
wrote:

> Hi,
>
> This query produces the following result:
>
> let $debates := collection("Parliament")
> for $debate-item in $debates
>let $item-file := $debate-item/officiele-publicatie//meta/@content
>let $debate-id := fn:analyze-string(
>  $debate-item/officiele-publicatie//meta/@content,
> "(\d{8}-\d*)-(\d*)")
>return ($debate-id)
>
> =>
>
> http://www.w3.org/2005/xpath-functions";>
>https://zoek.officielebekendmakingen.nl/h-tk-
>
>  20202021-102-1
>
>/metadata.xml
> 
> ...
>
> I am trying to extract the values from group 1 and 2 but this query
> returns 0 results:
>
> let $debates := collection("Parliament")
> for $debate-item in $debates
>let $item-file := $debate-item/officiele-publicatie//meta/@content
>let $debate-id := fn:analyze-string(
>  $debate-item/officiele-publicatie//meta/@content, "(\d{8}-\d*)-(\d*)")
>
>  let $debate-nr := $debate-id//match/group[@nr="1"]/text()
>  let $item-nr := $debate-id//match/group[@nr="2"]/text()
>
>return ($debate-nr, $item-nr)
>
> My guess is that analyze-string inserts new elements in the query and
> that that is the reason why this does not work.
>
> How can I extract debate-nr and item-nr from $debate-id?
>
> Ben Engbers
>

Re: [basex-talk] Managing/Debugging Server Load and Performance

[Bridger Dyson-Smith]

Hi Eliot -

On Mon, Feb 7, 2022 at 3:17 PM Eliot Kimber 
wrote:

> I like the multiple instance case—I hadn’t consider that, but if I
> undertand how BaseX works with indexes, it should be as easy as making a
> backup from instance A and restoring it to instance B?
>
>
>
> Or maybe just have a shared /data dir and be careful about which instance
> is accessing a given database so there are not concurrent writes?
>
> You are probably already familiar with the filesystem locks, but in case
you aren't, see [1]

>
>
> I’m using node IDs as index lookup keys so I’d have to make sure that node
> IDs persist across backup and restore but surely that is the case?
>
>
>
> But it certainly makes sense to do my long-running persistent index
> construction in one instance and then expose it through another as
> read-only for user access.
>
> I can't find it quickly, but I recall Marco Lettere writing about a
similar setup for some of his architecture. That may help when looking
through the list archive.

>
>
> Will explore this.
>
>
>
> Cheers,
>
>
>
> E.
>
>
>
Best,
Bridger

[1] https://docs.basex.org/wiki/Transaction_Management#File-System_Locks

> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>
>
>
> *From: *Christian Grün 
> *Date: *Monday, February 7, 2022 at 12:28 PM
> *To: *Eliot Kimber 
> *Cc: *basex-talk@mailman.uni-konstanz.de <
> basex-talk@mailman.uni-konstanz.de>
> *Subject: *Re: [basex-talk] Managing/Debugging Server Load and Performance
>
> [External Email]
>
>
> > I am seeing that when BaseX is doing some long-running process
> (constructing one of our big custom indexes) that it seems to consume all
> the BaseX cycles and the HTTP server becomes unresponsive.
>
> If you perform expensive write operations on databases that other
> users want to access, they’ll need to wait until this operation is
> finished.
>
> There are various ways out: You can e.g. create temporary instances of
> your index databases and drop the operative database instances after
> your maintenance operations have finished. For high-availability use
> cases, you can also work with multiple BaseX instances.
>
> > […] this use of BaseX was completely unplanned but suddenly very much
> desired…
>
> Always good to hear!
>

Re: [basex-talk] Configuration Option for Files to Parse in Create Database Dialog?

[Bridger Dyson-Smith]

Apologies - I hit send before including the link

On Thu, Jan 13, 2022 at 4:29 PM Bridger Dyson-Smith 
wrote:

> Hi Eliot,
>
> On Thu, Jan 13, 2022 at 3:54 PM Eliot Kimber 
> wrote:
>
>> I’m using the BaseX GUI in a very informal way where I create new
>> databases as needed to do ad-hoc queries over content that comes from
>> various git repos.
>>
>>
>>
>> I’m trying to setup my local configuration settings to be what I need for
>> this database creation tasks.
>>
>>
>>
> Are you editing a .basexgui file?
>
> I’ve found and successfully configured the options for skipping corrupt
>> and parsing archives but I don’t see an option that will populate the Input
>> file or directory field, which I need to be “*.xml, **.dita,**.ditamap”.
>>
>
> Does CREATEFILTER[1] help? I wonder if adding something like `CREATEFILTER
> = *.xml, *.dita, *.ditamap` would be sufficient - adding it at the end of
> the .basexgui file, after the `# Local Options` line?
> I'm not sure though - off the cuff suggestion are all I have at the moment.
>
>
>>
>> I found the option that sets the pattern used in the main file selector
>> field but that doesn’t apply to this dialog. I’m not seeing anything for
>> this in the docs.
>>
>>
>>
>> This is for BaseX 9.6.3
>>
>>
>>
>> Thanks,
>>
>>
>>
>> E.
>>
>> __
>>
>
> Hopefully this is helpful - apologies if not.
> Best,
> Bridger
>

[1]  https://docs.basex.org/wiki/Options#CREATEFILTER

>
> ___
>>
>> *Eliot Kimber*
>>
>> Sr Staff Content Engineer
>>
>> O: 512 554 9368
>>
>> M: 512 554 9368
>>
>> servicenow.com <https://www.servicenow.com>
>>
>> LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter
>> <https://twitter.com/servicenow> | YouTube
>> <https://www.youtube.com/user/servicenowinc> | Facebook
>> <https://www.facebook.com/servicenow>
>>
>

Re: [basex-talk] Configuration Option for Files to Parse in Create Database Dialog?

[Bridger Dyson-Smith]

Hi Eliot,

On Thu, Jan 13, 2022 at 3:54 PM Eliot Kimber 
wrote:

> I’m using the BaseX GUI in a very informal way where I create new
> databases as needed to do ad-hoc queries over content that comes from
> various git repos.
>
>
>
> I’m trying to setup my local configuration settings to be what I need for
> this database creation tasks.
>
>
>
Are you editing a .basexgui file?

I’ve found and successfully configured the options for skipping corrupt and
> parsing archives but I don’t see an option that will populate the Input
> file or directory field, which I need to be “*.xml, **.dita,**.ditamap”.
>

Does CREATEFILTER[1] help? I wonder if adding something like `CREATEFILTER
= *.xml, *.dita, *.ditamap` would be sufficient - adding it at the end of
the .basexgui file, after the `# Local Options` line?
I'm not sure though - off the cuff suggestion are all I have at the moment.


>
> I found the option that sets the pattern used in the main file selector
> field but that doesn’t apply to this dialog. I’m not seeing anything for
> this in the docs.
>
>
>
> This is for BaseX 9.6.3
>
>
>
> Thanks,
>
>
>
> E.
>
> __
>

Hopefully this is helpful - apologies if not.
Best,
Bridger

___
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>

Re: [basex-talk] Help dealing with slow API responses

[Bridger Dyson-Smith]

Hi Christian,

As always, thanks for your time and help.

On Wed, Nov 24, 2021 at 12:18 PM Christian Grün 
wrote:

> Hi Bridger,
>
> > I'm pulling data back from an OAI-PMH endpoint that is slow; i.e.
> response times are ~1/minute.
>
> I’ve tried the example you have attached (thanks). It’s seems to be
> much faster. Do you think that’s just my geographic proximity to the
> Konstanz-based server, or did you use a different setting for your
> slow tests?
>
> I think that it is partially geographic proximity and partly that the
system that has given me trouble is just incredibly slow; I'm hesitant to
share the particular URL.


> > 1. Is there a better way, using the BaseX GUI (or the command line), to
> get feedback on a querying process like this?
>
> If you use the BaseX GUI and if you restart a query or run a second
> one, the first one will be interrupted, so I guess you’ll have similar
> experiences with IntelliJ. But…
>
> > Something... asynchronous, or something clever with builtin functions in
> the `jobs` or `xquery` modules?
>
> You could create multiple query jobs, which run in parallel, with the
> jobs:eval function. They will only be interrupted if the IDE is
> stopped, but your IDE won’t notify you when the queries terminate
> normally or expectedly.
>
> A promising alternative for you could be xquery:fork-join [1]. In fact
> we mostly use it for running multiple slow HTTP requests in parallel:
>
> xquery:fork-join(
>   for $segment in 1 to 4
>   let $url := 'http://url.com/path/' || $segment
>   return function() { http:send-request((), $url) }
> )
>
> The function will terminate once all parallel requests have returned a
> response (and the results will be returned in the expected order).
>
> I've used `xquery:fork-join()` for something else in the past, and it is
truly fantastic; as you mention here
and in the documentation, it makes slower HTTP requests much easier. Maybe
I'm not thinking carefully
about my particular issue, but I don't know if using fork-join will help in
this case. The initial query to the API
returns some data; e.g.


  ...
  abc123


and the following queries rely on the existence (or lack) of the value in
example/token/text(). Those values are, AFAIK,
possibly randomized, or even just structured differently between the
various endpoints that I use, so I wouldn't be
able to know the full URLs to structure a fork-join.

I'm not sure I'm capturing my problem, but thanks for letting me talk it
through here.


> Next, you could run a script multiple times on command line and e.g.
> assign different arguments:
>
> > basex -bvar=1 query.xq
> > basex -bvar=2 query.xq
> > ...
>
> query.xq:
> declare variable $var external;
> file:write($var || '.xml', ...)
>
> > 2. If this can be addressed relatively directly with RESTXQ
>
> RESTXQ can be helpful if you write web applications, or if you want to
> define custom REST endpoints. It’s true that such endpoints can then
> be called multiple times as well, and will run in parallel as long as
> the queries don’t write to the same databases [2]. Maybe it’s overkill
> if you only want to run scripts in parallel, though. The more basic
> client/server architecture could be an alternative [3]; it can be used
> similar to the command line solution.
>
> I guess my thinking in regards to RESTXQ was that maybe, assuming I have
the
proper functions in place, I could return a new webpage while the following
function calls
were happening in the background; e.g.

step 1: start query to a given endpoint
step 2: when the first result is returned, redirect the user (me) to a new
webpage with a message (and the token; e.g. 'abc123'), and
step 3: using the token, launch the following query (which relies on said
token)
step 4: when the result is returned, redirect the user to a new webpage
with an updated message (and both tokens (first, and second); e.g. 'abc123'
and 'def456'),
step 5: etc until the process finishes.

Again, that's the RESTXQ that was happening in my imagination, but I'm
definitely just at the imagining phase with
this, so please excuse me if I'm misconstruing or just thinking about
things poorly! :)

> I've attached a simple SSCCE, where the basic idea is: query an API for
> some data, and get a response like so:
>
> You indicated that you are sending two requests. Is it the first one
> that’s slow? Does the first response create all input elements for the
> second requests, or do you have twice the number of requests in total?
>

In my real world case, which again I hesitate to share, *all* requests are
slow. In the meantime, maybe this new URL/endpoint might help illustrate.
Using the following
for $url and $verb (and apologies, my shell seems to mislike "&", hence
the "&" - that may need to change depending on your environment), the
initial response (ending with 500:7603::) is very quick to the terminal,
but the subsequent responses are built up and returned all at the same time.
$ basex -burl="http://dpla.lib.utk.edu/repox/OAIHa

[basex-talk] Help dealing with slow API responses

[Bridger Dyson-Smith]

Hi all -
I have the sneaking suspicion that the answer to this plea for help will be
something like, "Use RESTXQ!!" or something similar, but let me describe
the problem: I'm pulling data back from an OAI-PMH endpoint that is slow;
i.e. response times are ~1/minute. Embarrassingly, I think I've spent
several hours trying to figure out if my requests were buggy before I
realized that the API endpoint was just *slow* (at least compared to others
that I use regularly).

I typically use the IntelliJ plugin or the BaseX GUI for the vast majority
of my XQuery work, but these requests effectively build up a sequence of
responses (as files) and then serialize them all to disk when the requests
have finished. When the requests are answered quickly, there's prompt
feedback (my script finishes quickly, I can see serialized files, etc), but
when the requests are answered slowly, I'm left waiting (and then thinking,
"Oh no - I've mistyped the endpoint URL." or something similar).

Initially I think I have two requests for guidance:
1. Is there a better way, using the BaseX GUI (or the command line), to get
feedback on a querying process like this? Something... asynchronous, or
something clever with builtin functions in the `jobs` or `xquery` modules?
2. If this can be addressed relatively directly with RESTXQ, I'm game to
(finally) get more comfortable with it, but does anyone have any examples,
applications, scripts, etc they would be willing to share? I want to say
that several examples and ideas have been shared before here on the list,
but I'm having a terrible time finding them.

I've attached a simple SSCCE, where the basic idea is: query an API for
some data, and get a response like so:

  ...
  abc123

then take the text() of the resumptionToken, and resubmit a request to the
API, which would return:

  ...
  def456

The full responses are written to the temp directory on your system
(file:temp-dir()), with a date-stamp name.

The endpoint in my attached example has a fairly small response from what
I've selected, but the idea would be: how let me (or another user) know
that the query is active and running, not hung up or failing? I understand
that given the functional nature of XQuery these sorts of things can be a
bit more complicated, so I would appreciate any thoughts, opinions, links,
etc.

Many thanks for your time and trouble.
Best,
Bridger

PS the endpoint in the example doesn't necessarily exhibit the same slow
behavior that is the basis for my woes -- for some list readers it may be
very fast indeed -- but I felt like it might be apropos to use this
particular endpoint for an illustration.


quick-example.xq
Description: Binary data

Re: [basex-talk] Passing options to saxon through xslt:transform

[Bridger Dyson-Smith]

Graydon,
There's almost certainly a much nicer way to do this - eg the examples
you've provided - but could another option be to build out a
`proc:execute()` call? It wouldn't be nearly as elegant, and may cause
additional processing in the output, but that way you would have full
access to command line Saxon options and  switches.

Best, (and apologies for the water cooler suggestion!)
Bridger


On Mon, Nov 8, 2021, 6:11 PM Graydon Saunders  wrote:

> Hello --
>
> Somewhat to my surprise I've got the large XSLT 3.0 transform to load, or
> at least not complain about includes. (Load the input as a db; load the
> stylesheet from the file system. Be extra-careful about constructing
> URIs.)  Which raises the prospect that I'll actually be able to run this
> transform from BaseX, but I cannot figure out how to set a value for the
> initial template.
>
> This query:
>
> let $source := 
>
> let $template :=
> 
>   
> Hello, World!
>   
>
>   
> matched on root
>   
> 
>
> return xslt:transform($source,$template,map{'initial-template': '{
> http://www.w3.org/1999/XSL/Transform}initial-template'})
>
> returns
> matched on root
>
> Which isn't surprising; setting initial-template is a Saxon option, not a
> parameter.
>
> If I try
> return xslt:transform($source,$template,map{},map{'initial-template': '{
> http://www.w3.org/1999/XSL/Transform}initial-template'})
>
> I get [XPTY0004] Unknown option 'initial-template'. (same with 'it'.)
>
> Is there a way to set Saxon options with xslt:transform?
>
> Thanks!
> Graydon
>
>
>

Re: [basex-talk] basex test test.xqm => XPDY002c

[Bridger Dyson-Smith]

Jonathan - apologies for the truncated email.

This also works for me:
bridger@ditrestle|~/src/basex-src on master!
± basex -t basex-core/src/test/resources/test.xqm

  


  
Empty sequence.
  

...etc...

e.g. using the `-t` flag on the basex command.

I hope that's helpful.
Best,
Bridger

On Fri, Oct 22, 2021 at 4:00 PM Bridger Dyson-Smith 
wrote:

> Hi Jonathan -
>
> I needed to launch the command line basex; e.g.
> bridger@ditrestle|~/src/basex-src on master!
> ± basex
> BaseX 9.6.2 [Standalone]
> Try 'help' to get more information.
> > TEST basex-core/src/test/resources/test.xqm
> 
>name="file:///usr/home/bridger/src/basex-src/basex-core/src/test/resources/test.xqm"
> time="PT0.914S" tests="8" failures="4" errors="1" skipped="1">
> 
> 
>   
> Empty sequence.
>   
> 
> 
>   
> 9
> 6
> Item 1: 6 expected, 9 returned.
>   
> 
> 
>   
> err:FORG0001
>   
> 
> 
> 
>   
> Failure!
>   
> 
> 
>   
> Cannot convert to xs:double: .
>   
> 
> 
>   
> 
> Result: 8 tests, 4 failures, 1 error, 1 skipped.
>
>
>
> On Fri, Oct 22, 2021 at 3:37 PM Jonathan Robie 
> wrote:
>
>> I get this error with either the test.xqm found in the BaseX GitHub repo
>> or a copy of the file taken from the web page:
>>
>> % basex test test.xqm
>> Stopped at /Users/jonathan/github/basex/basex-core/src/test/resources/,
>> 1/5:
>> [XPDY0002] test: Context is undeclared.
>>
>> What am I doing wrong?
>>
>> Jonathan
>>
>

Re: [basex-talk] basex test test.xqm => XPDY002c

[Bridger Dyson-Smith]

Hi Jonathan -

I needed to launch the command line basex; e.g.
bridger@ditrestle|~/src/basex-src on master!
± basex
BaseX 9.6.2 [Standalone]
Try 'help' to get more information.
> TEST basex-core/src/test/resources/test.xqm

  


  
Empty sequence.
  


  
9
6
Item 1: 6 expected, 9 returned.
  


  
err:FORG0001
  



  
Failure!
  


  
Cannot convert to xs:double: .
  


  

Result: 8 tests, 4 failures, 1 error, 1 skipped.



On Fri, Oct 22, 2021 at 3:37 PM Jonathan Robie 
wrote:

> I get this error with either the test.xqm found in the BaseX GitHub repo
> or a copy of the file taken from the web page:
>
> % basex test test.xqm
> Stopped at /Users/jonathan/github/basex/basex-core/src/test/resources/,
> 1/5:
> [XPDY0002] test: Context is undeclared.
>
> What am I doing wrong?
>
> Jonathan
>

Re: [basex-talk] json serialization : how to ignore new lines and tabs ?

[Bridger Dyson-Smith]

As always, thank you Christian for helping clarify.

Tschüß!
Bridger

On Thu, Oct 21, 2021 at 11:04 AM Christian Grün 
wrote:

> Hi Bridger,
>
> Thanks for your valuable observation; you are completely right. I have
> opened a new issue for tracking down this bug [1].
>
> Best,
> Christian
>
> [1] https://github.com/BaseXdb/basex/issues/2044
>
>
> On Thu, Oct 21, 2021 at 4:49 PM Bridger Dyson-Smith 
> wrote:
>
>> Hi Maud -
>> with apologies, I thought I had added a namespace value to my example (I
>> clearly did not).
>>
>> Hi Christian - I'm a bit confused and I wonder if this is just the
>> typical user/operator error on my part, or if this is a bug? The BaseX
>> documentation for the JSON module[1] states that "... namespaces, comments,
>> and processing instructions will be discarded in the transformation
>> process.", but the output of the following still has namespaces:
>>
>> declare variable $test :=
>>   http://canofbees.org/ns/";>;
>> json:serialize($test, map { "format": "jsonml", "indent": false() })
>>
>> returns
>> ["me", {"xmlns:m":"http:\/\/canofbees.org\/ns\/"},["mine",
>> {"test":"false"}]]
>>
>> I checked the JsonML documentation[2] and it seemed to imply that only
>> XHTML-namespaced nodes would have their namespaces dropped, but that
>> doesn't seem to happen either; e.g.
>>
>> declare variable $xhtml :=
>>   http://www.w3.org/1999/xhtml";>
>> 
>>   element
>>   test
>> 
>>   ;
>> json:serialize($xhtml, map { "format": "jsonml", "indent": false() })
>>
>> returns
>> ["dl", {"xmlns":"http:\/\/www.w3.org
>> \/1999\/xhtml"},["dt",["span","element"],["dd","test"]]]
>>
>> Thanks for any thoughts you can share on this.
>> Best,
>> Bridger
>>
>> [1] https://docs.basex.org/wiki/JSON_Module#JsonML
>> [2] http://www.jsonml.org/xml/
>>
>>
>> On Thu, Oct 21, 2021 at 4:31 AM Christian Grün 
>> wrote:
>>
>>> Hi Maud,
>>>
>>> In the following lines of code, your input is first serialized as XML,
>>> and then converted back to XML with whitespaces chopped and namespaces
>>> stripped. The result will then be serialized to JSON:
>>>
>>> let $xml := fetch:xml(
>>>   serialize($docs),
>>>   map { 'stripns': true(), 'chop': true()
>>> })
>>> return json:serialize($xml, map { 'format': 'jsonml' })
>>>
>>> Here’s one more solution to remove whitespace-only text nodes before
>>> serializing the data:
>>>
>>> let $xml := $docs update {
>>>   delete nodes .//text()[normalize-space() = '']
>>> }
>>> return json:serialize...
>>>
>>> It’s harder that way to get rid of namespaces. You’d probably need to
>>> rebuild the XML node, as e.g. demonstrated in [1].
>>>
>>> Hope this helps – and I hope you’re doing fine,
>>> Christian
>>>
>>> [1]
>>> https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg13678.html
>>>
>>>
>>>
>>> On Wed, Oct 20, 2021 at 5:57 PM Maud Ingarao 
>>> wrote:
>>>
>>>> Dear all
>>>>
>>>> We need to convert some xml data to json to create a nice js
>>>> visualisation in our webapp
>>>>
>>>> The json:serialize works fine, except the fact that it creates empty
>>>> strings data due to \n, \s and \t
>>>>
>>>> We can't find the option(s) to ignore them... And we would like to
>>>> ignore the namespace attributes as well.
>>>>
>>>> Below is an example of the input and output we get with
>>>>
>>>>  let $docs := {db:open('rey',
>>>> 'inventaire-ead.xml')//*:c[.//*:scopecontent[@localtype='Lettre']]}
>>>>  return  json:serialize($docs, map{ 'format': 'jsonml' })
>>>>
>>>> Thanks a lot !
>>>>
>>>> Maud
>>>> Input
>>>>
>>>> 
>>>> 
>>>>>>> normal="1747-01-24"/>
>>>>Bre 3-2
>>>>
>

Re: [basex-talk] json serialization : how to ignore new lines and tabs ?

[Bridger Dyson-Smith]

Hi Maud -
with apologies, I thought I had added a namespace value to my example (I
clearly did not).

Hi Christian - I'm a bit confused and I wonder if this is just the typical
user/operator error on my part, or if this is a bug? The BaseX
documentation for the JSON module[1] states that "... namespaces, comments,
and processing instructions will be discarded in the transformation
process.", but the output of the following still has namespaces:

declare variable $test :=
  http://canofbees.org/ns/";>;
json:serialize($test, map { "format": "jsonml", "indent": false() })

returns
["me", {"xmlns:m":"http:\/\/canofbees.org\/ns\/"},["mine",
{"test":"false"}]]

I checked the JsonML documentation[2] and it seemed to imply that only
XHTML-namespaced nodes would have their namespaces dropped, but that
doesn't seem to happen either; e.g.

declare variable $xhtml :=
  http://www.w3.org/1999/xhtml";>

  element
  test

  ;
json:serialize($xhtml, map { "format": "jsonml", "indent": false() })

returns
["dl", {"xmlns":"http:\/\/www.w3.org
\/1999\/xhtml"},["dt",["span","element"],["dd","test"]]]

Thanks for any thoughts you can share on this.
Best,
Bridger

[1] https://docs.basex.org/wiki/JSON_Module#JsonML
[2] http://www.jsonml.org/xml/


On Thu, Oct 21, 2021 at 4:31 AM Christian Grün 
wrote:

> Hi Maud,
>
> In the following lines of code, your input is first serialized as XML, and
> then converted back to XML with whitespaces chopped and namespaces
> stripped. The result will then be serialized to JSON:
>
> let $xml := fetch:xml(
>   serialize($docs),
>   map { 'stripns': true(), 'chop': true()
> })
> return json:serialize($xml, map { 'format': 'jsonml' })
>
> Here’s one more solution to remove whitespace-only text nodes before
> serializing the data:
>
> let $xml := $docs update {
>   delete nodes .//text()[normalize-space() = '']
> }
> return json:serialize...
>
> It’s harder that way to get rid of namespaces. You’d probably need to
> rebuild the XML node, as e.g. demonstrated in [1].
>
> Hope this helps – and I hope you’re doing fine,
> Christian
>
> [1]
> https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg13678.html
>
>
>
> On Wed, Oct 20, 2021 at 5:57 PM Maud Ingarao 
> wrote:
>
>> Dear all
>>
>> We need to convert some xml data to json to create a nice js
>> visualisation in our webapp
>>
>> The json:serialize works fine, except the fact that it creates empty
>> strings data due to \n, \s and \t
>>
>> We can't find the option(s) to ignore them... And we would like to ignore
>> the namespace attributes as well.
>>
>> Below is an example of the input and output we get with
>>
>>  let $docs := {db:open('rey',
>> 'inventaire-ead.xml')//*:c[.//*:scopecontent[@localtype='Lettre']]}
>>  return  json:serialize($docs, map{ 'format': 'jsonml' })
>>
>> Thanks a lot !
>>
>> Maud
>> Input
>>
>> 
>> 
>>> normal="1747-01-24"/>
>>Bre 3-2
>>
>>   
>>  Amsterdam, UB, Bibliotheek der Vereeniging tot
>> Bevordering van de Belangen des Boekhandels
>>   
>>
>>
>>> physdescstructuredtype="materialtype">
>>   1
>>   feuillet
>>
>> 
>> 
>>> linktitle="expediteur" relationtype="functionrelation"/>
>>> linktitle="destinataire" relationtype="functionrelation"/>
>>> relationtype="functionrelation"/>
>>> relationtype="functionrelation"/>
>> 
>> 
>>
>>   
>>
>>
>>   
>>
>>
>>   
>>
>> 
>> 
>>
>>   oui
>>
>>
>>   
>>  Si vous ecrivés aujourdhui ou vendredi
>>   
>>
>> 
>> 
>>
>>   oui
>>
>>
>>   oui
>>
>>
>>   cette lettre a certainement transitée par
>> Bousquet
>>
>> 
>>  
>> Output
>>
>>   ["c", {"xmlns":"http:\/\/ead3.archivists.org\/schema\/" 
>> , 
>> "xmlns:xsi":"http:\/\/www.w3.org\/2001\/XMLSchema-instance" 
>> , 
>> "audience":"external", "id":"Rey17470124", "level":"item"},
>> "\n",
>> ["did",
>>   "\n   ",
>>   ["unitdate", {"certainty":"high", "label":"composition", 
>> "normal":"1747-01-24"}],
>>   "\n   ",
>>   ["unitid",
>> "Bre 3-2"],
>>   "\n   ",
>>   ["repository",
>> "\n  ",
>> ["corpname",
>>   "\n ",
>>   ["part",
>> 

Re: [basex-talk] json serialization : how to ignore new lines and tabs ?

[Bridger Dyson-Smith]

Hi Maud -

Adding `json:serialize($docs, map { "format": "jsonml", "indent": false()
})` seems to drop all of the whitespace and namespaces, too. I'm using
v9.6.2 if that helps. It looks like `"indent": true()` is a default for
serialization.

```
declare variable $input :=
  

 
 Bre 3-2
 

   Amsterdam, UB, Bibliotheek der Vereeniging tot Bevordering
van de Belangen des Boekhandels

 
 
 
1
feuillet
 


   
   
   
   


   
  
   
   
  
   
   
  
   


   
  oui
   
   
  
 Si vous ecrivés aujourdhui ou vendredi
  
   


   
  oui
   
   
  oui
   
   
  cette lettre a certainement transitée par Bousquet
   

   ;

json:serialize($input, map { "format": "jsonml", "indent": false() })
```

returns:
```
["c", {"audience":"external", "id":"Rey17470124",
"level":"item"},["did",["unitdate", {"certainty":"high",
"label":"composition", "normal":"1747-01-24"}],["unitid","Bre
3-2"],["repository",["corpname",["part","Amsterdam, UB, Bibliotheek der
Vereeniging tot Bevordering van de Belangen des
Boekhandels"]]],["didnote"],["physdescstructured", {"coverage":"whole",
"physdescstructuredtype":"materialtype"},["quantity","1"],["unittype","feuillet"]]],["relations",["relation",
{"href":"Polier de Bottens, Georges Nicolas", "linktitle":"expediteur",
"relationtype":"functionrelation"}],["relation", {"href":"Bousquet, Marc
Michel", "linktitle":"destinataire",
"relationtype":"functionrelation"}],["relation", {"href":"Lausanne",
"linktitle":"expedition", "relationtype":"functionrelation"}],["relation",
{"href":"Genève", "linktitle":"reception",
"relationtype":"functionrelation"}]],["relatedmaterial",["relatedmaterial",
{"localtype":"imprime"},["bibref"]],["relatedmaterial",
{"localtype":"edition"},["bibref"]],["relatedmaterial",
{"localtype":"bibliographie"},["bibref"]]],["scopecontent",
{"localtype":"Lettre"},["scopecontent",
{"localtype":"adresse"},["p","oui"]],["scopecontent",
{"localtype":"incipit"},["blockquote",["p","Si vous ecrivés aujourdhui ou
vendredi",["odd",["odd",
{"localtype":"autographe"},["p","oui"]],["odd",
{"localtype":"signature"},["p","oui"]],["odd",
{"localtype":"contient"},["p","cette lettre a certainement transitée par
Bousquet"
```

On Wed, Oct 20, 2021 at 11:57 AM Maud Ingarao 
wrote:

> Dear all
>
> We need to convert some xml data to json to create a nice js visualisation
> in our webapp
>
> The json:serialize works fine, except the fact that it creates empty
> strings data due to \n, \s and \t
>
> We can't find the option(s) to ignore them... And we would like to ignore
> the namespace attributes as well.
>
> Below is an example of the input and output we get with
>
>  let $docs := {db:open('rey',
> 'inventaire-ead.xml')//*:c[.//*:scopecontent[@localtype='Lettre']]}
>  return  json:serialize($docs, map{ 'format': 'jsonml' })
>
> Thanks a lot !
>
> Maud
> Input
>
> 
> 
> normal="1747-01-24"/>
>Bre 3-2
>
>   
>  Amsterdam, UB, Bibliotheek der Vereeniging tot
> Bevordering van de Belangen des Boekhandels
>   
>
>
> physdescstructuredtype="materialtype">
>   1
>   feuillet
>
> 
> 
> linktitle="expediteur" relationtype="functionrelation"/>
> linktitle="destinataire" relationtype="functionrelation"/>
> relationtype="functionrelation"/>
> relationtype="functionrelation"/>
> 
> 
>
>   
>
>
>   
>
>
>   
>
> 
> 
>
>   oui
>
>
>   
>  Si vous ecrivés aujourdhui ou vendredi
>   
>
> 
> 
>
>   oui
>
>
>   oui
>
>
>   cette lettre a certainement transitée par Bousquet
>
> 
>  
> Output
>
>   ["c", {"xmlns":"http:\/\/ead3.archivists.org\/schema\/" 
> , 
> "xmlns:xsi":"http:\/\/www.w3.org\/2001\/XMLSchema-instance" 
> , 
> "audience":"external", "id":"Rey17470124", "level":"item"},
> "\n",
> ["did",
>   "\n   ",
>   ["unitdate", {"certainty":"high", "label":"composition", 
> "normal":"1747-01-24"}],
>   "\n   ",
>   ["unitid",
> "Bre 3-2"],
>   "\

Re: [basex-talk] Importing TSV files with a command file

[Bridger Dyson-Smith]

Jonathan -
apologies for misreading your email! I'm not well-versed in the .bxs
COMMAND syntax, but maybe this is closer?
I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm
under-caffeinated :)

Another piece of uncertainty: can you pass variables from the .bxs to a
script you are RUNning? Maybe Christian can help us both understand some
things!

Best,
Bridger

test-tsv.bxs
```
CREATE DB test-tsv
RUN /home/bridger/test-tsv.xq
LIST test-tsv
```
test-tsv.xq
```
for $T in file:children("/home/bridger/tsvs/")
let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab",
"header": true() })
let $TSV-NAME := file:name($T)
return db:add("test-tsv", $TSV, "/" || $TSV-NAME)
```
note: you could specify a db path here; e.g. db:add("test-tsv", $TSV,
"/tsv-files/" || $TSV-NAME)

I've attached my two sample TSV files if that helps.


On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie 
wrote:

> Thanks!  I was getting hung up trying to do this in a command.
>
> Jonathan
>
> On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie 
> wrote:
>
>>
>>
>> On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <
>> bdysonsm...@gmail.com> wrote:
>>
>>> Hi Jonathan,
>>>
>>> for $T in file:list($path-to-your-files)
>>> let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(),
>>> 'format': 'direct' })
>>> return db:add($your-db, $TSV)
>>>
>>> maybe? I'm probably getting something wrong, semantics-wise, w/
>>> `db:add()`, but maybe it's close to get started?
>>> Best,
>>> Bridger
>>>
>>> On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie 
>>> wrote:
>>>
>>>> I have a command file that configures a database, throwing a bunch of
>>>> sources into various paths.
>>>>
>>>> I have a set of TSV files. I want to do the equivalent of ADD TO,
>>>> pointing to a directory of TSV files, so that they are parsed and converted
>>>> to XML and placed in the appropriate path.
>>>>
>>>> How do I do that?
>>>>
>>>> Jonathan
>>>>
>>>
ID  EGF_BaselineEGF_Stimulus
FBgn0029994 -1.25   -0.27
FBgn0037191 -1.05   0.78
FBgn0036810 2.081.34
FBgn0033320 1.150.45
FBgn0051156 -1.77   -0.76
FBgn0004635 -0.77   -0.42
FBgn0260646 -0.92   0.97
FBgn0038397 0.040.22
FBgn0010583 -0.77   -0.59

test-tsv.bxs
Description: Binary data


test-tsv.xq
Description: Binary data
nameflight  rsvp
alice   TYS234  yes
billFLL982  no
clarice PHL4221 yes
devon   BNA740  yes

Re: [basex-talk] Importing TSV files with a command file

[Bridger Dyson-Smith]

Hi Jonathan,

for $T in file:list($path-to-your-files)
let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(),
'format': 'direct' })
return db:add($your-db, $TSV)

maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`,
but maybe it's close to get started?
Best,
Bridger

On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie 
wrote:

> I have a command file that configures a database, throwing a bunch of
> sources into various paths.
>
> I have a set of TSV files. I want to do the equivalent of ADD TO, pointing
> to a directory of TSV files, so that they are parsed and converted to XML
> and placed in the appropriate path.
>
> How do I do that?
>
> Jonathan
>

Re: [basex-talk] Problems installing EXPath

[Bridger Dyson-Smith]

Ah, right - Christian and team added the Functx library as a default... at
some point between 9.1 and now. 😂

Glad that helped.
Have a pleasant weekend.

On Fri, Sep 3, 2021 at 3:39 PM Jonathan Robie 
wrote:

> This is BaseX 9.1.   When I do a repo list, it does not list EXPath.
>
> Using https fixed the problem - thanks!
>
> Jonathan
>
> On Fri, Sep 3, 2021 at 1:58 PM Bridger Dyson-Smith 
> wrote:
>
>> Hi Jonathan -
>>
>> I'm curious: which distribution of BaseX are you using? The main
>> BaseX.zip has the functx library already in ${basex}/repo.
>>
>> I did though do a quick test (below) and keeping https in the URL made a
>> difference. I'm not quite sure why.
>> Hope that's helpful.
>> Best,
>> Bridger
>> ```
>> > REPO INSTALL https://files.basex.org/modules/expath/functx-1.0.xar
>> Package 'https://files.basex.org/modules/expath/functx-1.0.xar' replaced
>> in 1158.39 ms.
>> > REPO
>> Stopped at , 1/5:
>> Syntax: REPO [DELETE|INSTALL|LIST]
>>   Install, delete or list packages.
>>
>> Installs, deletes or lists packages from the repository.
>> - DELETE [name|dir]:
>>   deletes the package with name [name] or directory [dir]
>> - INSTALL [path]:
>>   installs package with path [path]
>> - LIST:
>>   lists all installed packages
>> > REPO LIST
>> Name   Version  TypePath
>> ---
>> http://www.functx.com  1.0  EXPath  http-www.functx.com-1.0
>>
>> 1 package(s).
>> > REPO DELETE http://www.functx.com
>> Package 'http://www.functx.com' deleted.
>> > REPO LIST
>> Name  Version  Type  Path
>> -
>>
>> 0 package(s).
>> > REPO INSTALL http://files.basex.org/modules/expath/functx-1.0.xar
>> Stopped at , 1/66:
>> [repo:parse] functx-1.0.xar: Resource "expath-pkg.xml" not found..
>> >
>> ```
>>
>> On Thu, Sep 2, 2021 at 8:59 PM Jonathan Robie 
>> wrote:
>>
>>> I tried installing EXPath, but without success.
>>>
>>>
>>> > REPO INSTALL http://files.basex.org/modules/expath/functx-1.0.xar
>>> Stopped at , 1/66:
>>> [repo:parse] functx-1.0.xar: Resource "expath-pkg.xml" not found..
>>>
>>>
>>> If I download the .xar and unzip it, it looks like this:
>>>
>>>
>>> [image: image.png]
>>>
>>> Here is the content of expath-pkg.xml:
>>>
>>>
>>> http://expath.org/ns/pkg";
>>>  name="http://www.functx.com";
>>>  abbrev="functx"
>>>  version="1.0"
>>>  spec="1.0">
>>>
>>>FunctX library
>>>
>>>
>>>   http://www.functx.com/functx.xsl
>>>   functx.xsl
>>>
>>>
>>>   http://www.functx.com
>>>   functx.xq
>>>
>>>
>>> 
>>>
>>> Any thoughts?
>>>
>>>
>>> Jonathan
>>>
>>>

Re: [basex-talk] Problems installing EXPath

[Bridger Dyson-Smith]

Hi Jonathan -

I'm curious: which distribution of BaseX are you using? The main BaseX.zip
has the functx library already in ${basex}/repo.

I did though do a quick test (below) and keeping https in the URL made a
difference. I'm not quite sure why.
Hope that's helpful.
Best,
Bridger
```
> REPO INSTALL https://files.basex.org/modules/expath/functx-1.0.xar
Package 'https://files.basex.org/modules/expath/functx-1.0.xar' replaced in
1158.39 ms.
> REPO
Stopped at , 1/5:
Syntax: REPO [DELETE|INSTALL|LIST]
  Install, delete or list packages.

Installs, deletes or lists packages from the repository.
- DELETE [name|dir]:
  deletes the package with name [name] or directory [dir]
- INSTALL [path]:
  installs package with path [path]
- LIST:
  lists all installed packages
> REPO LIST
Name   Version  TypePath
---
http://www.functx.com  1.0  EXPath  http-www.functx.com-1.0

1 package(s).
> REPO DELETE http://www.functx.com
Package 'http://www.functx.com' deleted.
> REPO LIST
Name  Version  Type  Path
-

0 package(s).
> REPO INSTALL http://files.basex.org/modules/expath/functx-1.0.xar
Stopped at , 1/66:
[repo:parse] functx-1.0.xar: Resource "expath-pkg.xml" not found..
>
```

On Thu, Sep 2, 2021 at 8:59 PM Jonathan Robie 
wrote:

> I tried installing EXPath, but without success.
>
>
> > REPO INSTALL http://files.basex.org/modules/expath/functx-1.0.xar
> Stopped at , 1/66:
> [repo:parse] functx-1.0.xar: Resource "expath-pkg.xml" not found..
>
>
> If I download the .xar and unzip it, it looks like this:
>
>
> [image: image.png]
>
> Here is the content of expath-pkg.xml:
>
>
> http://expath.org/ns/pkg";
>  name="http://www.functx.com";
>  abbrev="functx"
>  version="1.0"
>  spec="1.0">
>
>FunctX library
>
>
>   http://www.functx.com/functx.xsl
>   functx.xsl
>
>
>   http://www.functx.com
>   functx.xq
>
>
> 
>
> Any thoughts?
>
>
> Jonathan
>
>

Re: [basex-talk] BaseX 9.6: The Summer Edition

[Bridger Dyson-Smith]

Dear Christian and everyone else on the BaseX team:
As always, thank you all so much for your time and effort!
Looking forward to 9.6!
Best,
Bridger

On Thu, Aug 19, 2021 at 8:22 AM Christian Grün 
wrote:

> Dear all,
>
> We provide you with a new and fresh version of BaseX, our open source
> XML framework, database system and XQuery 3.1 processor:
>
>   https://basex.org/
>
> Apart from our main focus (query rewritings and optimizations), we
> have added the following enhancements:
>
> XQUERY: MODULES, FEATURES
> - Archive Module, archive:write: stream large archives to file
> - SQL Module: support for more SQL types
> - Full-Text Module, ft:thesaurus: perform Thesaurus queries
> - Fulltext, fuzzy search: specify Levenshtein limit
> - UNROLLLIMIT option: control limit for unrolling loops
>
> XQUERY: JAVA BINDINGS
> - Java objects of unknown type are wrapped into function items
> - results of constructor calls are returned as function items
> - the standard package "java.lang." has become optional
> - array arguments can be specified with the middle dot notation
> - conversion can be controlled with the WRAPJAVA option
> - better support for XQuery arrays and maps
>
> WEB APPLICATIONS
> - RESTXQ: Server-Timing HTTP headers are attached to the response
>
> For a more comprehensive list of added and updated features, look into
> our documentation (docs.basex.org) and check out the GitHub issues
> (github.com/BaseXdb/basex/issues).
>
> Have fun,
> Your BaseX Team
>

[basex-talk] Copy/modify, Update, or file:append - help appreciated

[Bridger Dyson-Smith]

Hi all -
I'm trying to keep track of some HTTP POST results in a file, something
akin to the following:

file:write-text(
  "/tmp/bridger-test.csv",
  "book" || "," || "title" || "," || "sip" || out:nl()
),

for $i in 1 to 20
(: some http:send-request stuff :)
return(
  prof:sleep(250),
  if ($request[1]//@status/data() = 201)
  then file:append(
"/tmp/bridger-test.csv",
"book: " $i || "," || "title: " $i || "," || ""
  )
  else "something went wrong with the request; the response was: " ||
$request[1]//@status/data()

I have something that kinda works, but I'd like it to be better XQuery,
avoid the two step create-file-then-append, and leverage more of built-in
awesomeness in the language. I'm sure that I need to have a better
understanding of the Update syntax, and/or %updating annotation, but would
someone be willing to give some hints and/or suggestions for a better way
to handle situations like this?
Thanks in advance for your time and trouble.
Best,
Bridger

Re: [basex-talk] BaseX with Apache

[Bridger Dyson-Smith]

Hi Tim -

What about modifying the BaseX web.xml in $basex/webapp/WEB-INF? That was
you'd be disabling in Jetty instead of Apache. To be honest I have no idea
if this will work, but:


  default
  org.eclipse.jetty.servlet.DefaultServlet
  
dirAllowed
false
  

 or some variation thereof? I cribbed this from stackoverflow, naturally
(and here's where I found it: [1])

Hope that's helpful, apologies if it's only noise.
Best,
Bridger

[1]
https://stackoverflow.com/questions/7217969/how-to-disable-directory-listing-for-jettys-webappcontext

On Thu, May 27, 2021 at 9:48 AM Tim Thompson  wrote:

> With apologies for the not-strictly-BaseX question--but I'm running BaseX
> through a reverse proxy with Apache (and Let's Encrypt), and I can't figure
> out how to disable directory listing for the BaseX webapp/static directory.
> I've tried editing the apache2 conf files in the standard ways. Hoping
> someone here has a quick solution to share.
>
> Thanks in advance,
> Tim
>
> --
> Tim A. Thompson
> Metadata Librarian
> Yale University Library
>
>

Re: [basex-talk] A question about the arrow operator and predicates

[Bridger Dyson-Smith]

Christian -

As always, thank you so much!

On Wed, Mar 31, 2021 at 10:35 AM Christian Grün 
wrote:

> Dear Bridger,
>
> > I was wondering if anyone would have an insight for me as to why the
> following expression is wrong:
> >
> > for $s in ("/a/b/c", "/1/2/3")
> > let $t := $s => tokenize("/")[last()]
> > return $t
>
> This is due to the grammar rules of XQuery 3.1, which mandate that
> “=>” is followed by an “ArrowFunctionSpecifier” and an “ArgumentList”.
> Here are some production rules from the spec:
>
> [96] ArrowExpr ::= UnaryExpr ( "=>" ArrowFunctionSpecifier ArgumentList )*
> [127] ArrowFunctionSpecifier ::= EQName | VarRef | ParenthesizedExpr
> [122] ArgumentList ::= "(" (Argument ("," Argument)*)? ")"
> …
>
> The arrow function specifier can be an EQName, a variable reference or
> a parenthesized expression. The last one will do the job:
>
> for $s in ("/a/b/c", "/1/2/3")
> let $t := ($s => tokenize("/"))[last()]
> return $t
>
> It does - I had read some other online notes about parenthesizing part of
the left hand side expression, but hadn't parenthesized properly.
As a follow up, for my layman's thinking about XPath/XQuery, is this
effectively creating a sequence that is then filtered? Or am I mentally
overloading the parentheses?

Thanks for the insight!

Hope this helps,
> Christian
>
> [1] https://www.w3.org/TR/xquery-31/#id-arrow-operator
>
>
Best,
Bridger

>
>
>
> >
> > Thanks for your help!
> > Best,
> > Bridger
> >
> > PS I don't always remember to look at the optimized query in the Info
> window, but when I do I always get a hint about something; "util:last()" in
> this case.
> >
> > correctly
> > ```
> > for $s in ("/a/b/c", "/1/2/3")
> > let $t := tokenize($s, "/")[last()]
> > return $t
> > ```
> > ```
> > for $s in ("/a/b/c", "/1/2/3")
> > let $t := $s => tokenize("/") => util:last()
> > return $t
> > ```
>

[basex-talk] A question about the arrow operator and predicates

[Bridger Dyson-Smith]

Hi all -

I was wondering if anyone would have an insight for me as to why the
following expression is wrong:

```
for $s in ("/a/b/c", "/1/2/3")
let $t := $s => tokenize("/")[last()]
return $t
```
This is because of the predicate filter, but I'm not clear on *why* it's
because of that :)

Thanks for your help!
Best,
Bridger

PS I don't always remember to look at the optimized query in the Info
window, but when I do I always get a hint about something; "util:last()" in
this case.

correctly
```
for $s in ("/a/b/c", "/1/2/3")
let $t := tokenize($s, "/")[last()]
return $t
```
```
for $s in ("/a/b/c", "/1/2/3")
let $t := $s => tokenize("/") => util:last()
return $t
```

[basex-talk] Call for install/setup stories from users

[Bridger Dyson-Smith]

Hi all -

Per the recent thread about installing, I was hoping to convince some of
you to share your experiences installing and running BaseX. Whether you use
Mac OS, Windows, a Linux, or something else: how are you installing and
running BaseX?

I'm asking as there could be some possible improvements to the wiki that
might help new users (or even less-new users!) get up and running.

Thanks in advance for your time and trouble! If I can hear about number of
approaches, I'm happy to attempt writing up some details for the wiki.
Best,
Bridger

PS Here's mine:

Operating systems: I'm typically using BaseX on either a FreeBSD or a Void
Linux system (rarely on other Linuxes, and only once or twice on Windows).

Install process: my OS' don't have a pre-packaged download for BaseX, so I
do something like:
1. cd ~/bin
2. mkdir basex-src basex-data
3. fetch|wget https://files.basex.org/releases/BaseX.zip
4. unzip BaseX.zip
5. rm -rf basex/src basex/data
6. ln -s ~/bin/basex-src ~/bin/basex/src; ln -s ~/bin/basex-data
~/bin/basex/data
...
7. enjoy basex!

Upgrade process: something like
1. cd ~/bin
2. fetch|wget https://files.basex.org/releases/BaseX.zip
3. mv basex basex-old
4. unzip BaseX.zip
5. rm -rf basex/src basex/data
6. ln -s ~/bin/basex-src ~/bin/basex/src; ln -s ~/bin/basex-data
~/bin/basex/data
...

Usage note: I have a line in my shell RC file (~/.zshrc) that helps my
shell/command line know where to find the various basex executables:
`export PATH="$PATH:$HOME/bin/basex/bin"`

Re: [basex-talk] Preferences do not open, Exception in thread AWT-EventQueue-0

[Bridger Dyson-Smith]

On Wed, Mar 17, 2021 at 2:06 PM Joris Lambrecht 
wrote:

> Thanks for the perspective. Here's what i hope comes across as a polite
> rant as a reply. It aches me and i notice it aches others everywhere.
>
> I spent 20 years on and off with Linux, starting with a CT magazine CD
> somewhere late 90's. I've really grown stone cold tired of the package
> management nonsense with unmaintained, poorly maintained software.
> Except for the big packges like desktops that is for which no sane
> compilation procudure exists.
>
> I can appreciate this, but I'm always remembering that many of these
pieces of software are being maintained gratis, without any expectation of
recompensense (or thanks, even!). That said, you are not wrong - managing
widely-available downloads is a crazy process!

Also get irritated with the so called distro specific narrative. There
> are exactly 2 systems, Debian and RedHat of which Debian is universally
> compatible and Redhat is not so much. It should not be an issue really.
> This implies developing compatible with the RedHat system is alwasy
> going to work on Debian, not vice versa.
>
> I only use Redhat for work, but every time it's an exercise in remembering
"Is it yum? Is it rpm?" and I am sympathetic.  That being said, and maybe
I'm a different sort of outlier here, but there are a bunch of excellent
operating systems, only a few of which derive from Linux. They all handle
things differently, too, to be fair.

When it comes to package managers almost none of the work well or if
> they work well there is no sensible package maintenance happening. Or
> they build a great set of software like Sabayon Linux and stop the
> disro.
>
> The times i found myself downloading source and compiling from scratch
> to have all features and flags enabled became too many. Yet every time
> i go back almost to not using package managers. Such as I end up with
> now with BaseX 🙂
>
> By now i'm contemplating a no-package manager custom Linux distroy with
> update scripts for my personal choice of tools. Simply to avoid the
> cruft and 99% meaningless software in most distributions. Maybe i build
> it all on BaseX to manage configuration, compilation and other stuff.
>
> I'm a niche user who also happens to be picky and mostly atypical as an
> IT profile. The future will probably bring hard choices and hard work
> for me because of it.
>
> That resonates! Good luck (and welcome to BaseX)

>
> Br,
>
> Joris
>
>
Bridger

> On Wed, 2021-03-17 at 13:47 -0400, Bridger Dyson-Smith wrote:
> > Joris -
> >
> > On Wed, Mar 17, 2021 at 1:40 PM commandline-be
> >  wrote:
> > > Thanks for this.
> > >
> > >
> >
> > No problem and my pleasure.
> > > Basically I am pondering why I should use a packaged version and
> > > why not.
> > >
> >
> >
> > Why: I think there's an advantage to having a OS-based package that's
> > readily available - it's certainly very convenient to say `sudo yum
> > install basex` or `pkg install basex` or whatever your package
> > management expression happens to be, and whammo! you have a system-
> > wide BaseX installation.
> >
> > Why Not:
> > * someone has to maintain the package! Keep it up to date with minor
> > version bumps (one of the very nicest things about BaseX is how
> > quickly Christian and Co. respond to bugs and errors - minor bug
> > patches are infrequent but when there's an issue, there's a fix very
> > quickly), and modifying the default file paths to match your OS'
> > expectations.
> > * do you really need it installed system-wide? I've found that, at
> > least for my use-cases, an install to my home directory is sufficient
> > - but I'm not doing anything complicated! It would be great to hear
> > other user stories about this, which reminds me of a different email
> > I need to send to this list!
> >
> > >
> > > I did use the most recent .jar to test and it works well but the
> > > what about the server part etc.
> > >
> > >
> >
> > If you grab the ZIP archive, everything you'd want is included (http
> > server, GUI, client/server).
> >
> > > Br
> > >
> > > Joris
> > >
> > >
> >
> > Best,
> > Bridger
> > >
> > >
> > >
> > >
> > >
> > >  Oorspronkelijk bericht 
> > > Aan 17 mrt. 2021 18:07, Bridger Dyson-Smith <
> > > bdysonsm...@gmail.com> schreef:
> > > >
> > > > Hi Joris,
> > > &

Re: [basex-talk] Preferences do not open, Exception in thread AWT-EventQueue-0

[Bridger Dyson-Smith]

Joris -

On Wed, Mar 17, 2021 at 1:40 PM commandline-be 
wrote:

> Thanks for this.
>
> No problem and my pleasure.

> Basically I am pondering why I should use a packaged version and why not.
>

Why: I think there's an advantage to having a OS-based package that's
readily available - it's certainly very convenient to say `sudo yum install
basex` or `pkg install basex` or whatever your package management
expression happens to be, and whammo! you have a system-wide BaseX
installation.

Why Not:
* someone has to maintain the package! Keep it up to date with minor
version bumps (one of the very nicest things about BaseX is how quickly
Christian and Co. respond to bugs and errors - minor bug patches are
infrequent but when there's an issue, there's a fix very quickly), and
modifying the default file paths to match your OS' expectations.
* do you really need it installed system-wide? I've found that, at least
for my use-cases, an install to my home directory is sufficient - but I'm
not doing anything complicated! It would be great to hear other user
stories about this, which reminds me of a different email I need to send to
this list!


> I did use the most recent .jar to test and it works well but the what
> about the server part etc.
>
> If you grab the ZIP archive, everything you'd want is included (http
server, GUI, client/server).


> Br
>
> Joris
>
> Best,
Bridger

>
>
>
>
>
>  Oorspronkelijk bericht 
> Aan 17 mrt. 2021 18:07, Bridger Dyson-Smith < bdysonsm...@gmail.com>
> schreef:
>
>
> Hi Joris,
>
> On Wed, Mar 17, 2021 at 5:39 AM commandline-be 
> wrote:
>
>> ok, thanks.
>>
>>
>> If i can i will try and figure out an upgrade approach or see if i can
>> run a backport which does have the more recent version available.
>>
>> I've found that something like this following works very well for me
> across several operating systems:
>
> ```
> cd ~/bin
> mkdir basex-src basex-data
> wget https://files.basex.org/releases/BaseX.zip
> unzip BaseX.zip
> rm -rf basex/src basex/data
> ln -s ~/bin/basex-src ~/bin/basex/src; ln -s ~/bin/basex-data
> ~/bin/basex-data
> ```
> You can include the ~/bin/basex directory path in your environmental $PATH
> and you're off to the races.
>
> Subsequent updates are basically grabbing the ZIP archive, unpacking it,
> removing the default src and data directories, and recreating the symbolic
> links.
> All of your database info is kept separately from the defaults, so you
> don't worry about overwriting in an upgrade.
>
> It's still manual and necessitates some steps, but it's been an easy
> method for me across several different unix-like operating systems (Redhat,
> Void, and FreeBSD) that don't have a package for installation.
>
> I know some other people have posted similar approaches here on the
> mailing list, but I can't think of an easy search term to help locate them.
>
>>
>> Essentialy, i dislike Ubuntu a lot and Arch well, i never got round to
>> arch really.
>>
>> Best,
>>
>> Joris
>>
>> HTH
> Best,
> Bridger
>
>
>>
>> - - -
>> mailto:commandl...@protonmail.com
>>
>> ‐‐‐ Original Message ‐‐‐
>> On Wednesday, 17 March 2021 10:35, Christian Grün <
>> christian.gr...@gmail.com> wrote:
>>
>> > > Linux is eays, everything is a file. It are the specifics which make
>> it hard, particularly configuration.
>> >
>>
>> > Right. At the moment, there is no automatized process to get the
>> > Debian/Ubuntu distribution updates automatized.
>> >
>>
>> > > I think my issue with the Preferences panel may come from running the
>> most recent basex and it overwriting configuration files since it writes a
>> message to screen saying it overwrites the configuration file.
>> >
>>
>> > Thanks for the insight. If you want to stick with the old version of
>> > BaseX, you can delete the .basexgui file (which includes the GUI
>> > configuration) to resolve the issue.
>>
>>

Re: [basex-talk] Preferences do not open, Exception in thread AWT-EventQueue-0

[Bridger Dyson-Smith]

Hi Joris,

On Wed, Mar 17, 2021 at 5:39 AM commandline-be 
wrote:

> ok, thanks.
>
>
> If i can i will try and figure out an upgrade approach or see if i can run
> a backport which does have the more recent version available.
>
> I've found that something like this following works very well for me
across several operating systems:

```
cd ~/bin
mkdir basex-src basex-data
wget https://files.basex.org/releases/BaseX.zip
unzip BaseX.zip
rm -rf basex/src basex/data
ln -s ~/bin/basex-src ~/bin/basex/src; ln -s ~/bin/basex-data
~/bin/basex-data
```
You can include the ~/bin/basex directory path in your environmental $PATH
and you're off to the races.

Subsequent updates are basically grabbing the ZIP archive, unpacking it,
removing the default src and data directories, and recreating the symbolic
links.
All of your database info is kept separately from the defaults, so you
don't worry about overwriting in an upgrade.

It's still manual and necessitates some steps, but it's been an easy method
for me across several different unix-like operating systems (Redhat, Void,
and FreeBSD) that don't have a package for installation.

I know some other people have posted similar approaches here on the mailing
list, but I can't think of an easy search term to help locate them.

>
> Essentialy, i dislike Ubuntu a lot and Arch well, i never got round to
> arch really.
>
> Best,
>
> Joris
>
> HTH
Best,
Bridger


>
> - - -
> mailto:commandl...@protonmail.com
>
> ‐‐‐ Original Message ‐‐‐
> On Wednesday, 17 March 2021 10:35, Christian Grün <
> christian.gr...@gmail.com> wrote:
>
> > > Linux is eays, everything is a file. It are the specifics which make
> it hard, particularly configuration.
> >
>
> > Right. At the moment, there is no automatized process to get the
> > Debian/Ubuntu distribution updates automatized.
> >
>
> > > I think my issue with the Preferences panel may come from running the
> most recent basex and it overwriting configuration files since it writes a
> message to screen saying it overwrites the configuration file.
> >
>
> > Thanks for the insight. If you want to stick with the old version of
> > BaseX, you can delete the .basexgui file (which includes the GUI
> > configuration) to resolve the issue.
>
>

[basex-talk] Dr. Mike Kay's proposals for XPath and XQuery 4.0

[Bridger Dyson-Smith]

Hi all -

I imagine that Dr. Kay has posted about this in multiple places, but I
don't know if he's posting on the BaseX list much. He shared the following
links with his thoughts on XPath, XSLT, XQuery, and F&O version 4.0 [1] - I
don't know how many implementers we have on this list, outside of Christian
and co., but thought you all might find it interesting.

Hope everyone is having a lovely day.
Best,
Bridger

PS I found this announcement on the XML.com Slack channel, fyi.

[1] Mike Kay's post from Slack:
I've been working on fleshing out my ideas for XSLT/XPath 4.0, based on my
XML Prague paper earlier this year. I'm hoping to gather together a
community group of some kind to take this forward; meanwhile I've published
a very preliminary set of drafts:
XSLT: https://www.saxonica.com/qt4specs/XT/Overview-diff.html
F&O: https://www.saxonica.com/qt4specs/FO/Overview-diff.html
XPath: https://www.saxonica.com/qt4specs/XP/xpath-40-diff.html
XQuery: https://www.saxonica.com/qt4specs/XQ/xquery-40-diff.html
Preparing for the rotten tomatoes: let me know what you think and whether
you want to participate.
NOTE: each document has a summary of changes at the end. It's probably best
to go there first.

Re: [basex-talk] BaseX GUI and remote editing

[Bridger Dyson-Smith]

Hi Marco,

I'm sure others will have better thoughts on this, so please take this with
a grain of salt. Jetbrains IntelliJ (and other IDEs) provide Remote
Deployment options, and I would think Eclipse would offer something
similar. I.e. write locally, and then push changes to the remote server
with a keyboard chord. However, these types of connections require SSH or
something similar.

I don't know if that's a helpful thought or not.
Best,
Bridger

PS Shameless plug from a happy user: the Jetbrains IDEs can leverage Reece
Dunn's awesome XQuery plugin.

On Fri, Nov 13, 2020 at 9:50 AM Marco Lettere  wrote:

> Hello all!
>
> We are in the slightly uncomfortable condition to do some remote coding.
> Means write XQ code that has to be kept on a server different from our PCs.
>
> We don't want to slow down our very agile development workflow by
> passing everytime from a Git repo ...
>
> Has anyone here some experience with remote editing of this kind? Using
> BaseX Gui is out of discussion, isn't it?
>
> Thanks,
>
> Marco.
>
>

Re: [basex-talk] Help with a function to return paths

[Bridger Dyson-Smith]

Andy, et al,

Thank you for your examples (and apologies - I naturally managed to
copy/paste the wrong function into my initial email ).

Higher order functions are, sadly, a bit outside of my wheelhouse, so I'm
unable to fully appreciate their power and utility. But! happily I could
work from your examples and get something that I can almost wrap my head
around! Thank you so much!

Here's the final working example for me, I think:
```
declare function local:pb(
  $result as xs:string,
  $node as node()
) {
  concat(
$result, "/"[$result], name($node),
$node/@* ! local:atty(.) => string-join("")
  )
};

declare function local:atty(
  $att as attribute()
) as xs:string* {
  "[@" || name($att) || "='" || data($att) || "']"
};

distinct-values(
  $input//* ! fold-left(ancestor-or-self::*, '', local:pb(?, ?))
)
```
Again, I really appreciate the guidance!
Thanks and have a pleasant weekend,
Best,
Bridger


On Fri, Oct 16, 2020 at 1:01 PM Andy Bunce  wrote:

> If you really don't want the leading /
>
>  declare function local:pathbuild($result as xs:string,$this as node()){
>concat(
> $result, '/'[$result], name($this),
> $this/@*! concat("[@" , name(.) , "='" , data(.),
> "']")=>string-join('')
>   )
> };
>
> On Fri, 16 Oct 2020 at 17:48, Andy Bunce  wrote:
>
>> Hi Bridger,
>>
>> How about..
>>
>> declare function local:pathbuild($result as xs:string,$this as node()){
>>  concat(
>> $result, "/" , name($this),
>>     $this/@*! concat("[@" , name(.) , "='" , data(.),
>> "']")=>string-join('')
>>   )
>> };
>>
>> innermost($input//*)!fold-left(ancestor-or-self::*,'',local:pathbuild#2)
>>
>> /Andy
>>
>> On Fri, 16 Oct 2020 at 16:29, Bridger Dyson-Smith 
>> wrote:
>>
>>> Hi all -
>>>
>>> I hope you don't mind a question about serializing distinct XPaths. I'm
>>> trying to generate some reports for documentation and the available
>>> built-in and library functions[1] aren't quite what I need.
>>>
>>> The output I'm after is:
>>> test/aaa/bbb[@type="foo"]
>>> test/aaa/bbb[@type="foo"][@enc="bar"]
>>> test/aaa/bbb[@type="bzz"][@enc="bar"]
>>> test/aaa/bbb[@type="qux"][@enc="bar"][@key="yes"][@point="start"]
>>> test/aaa/bbb[@type="qux"][@enc="bar"][@key="yes"][@point="end"]
>>>
>>> I have a couple of functions that are getting me close, but I can't
>>> quite manage the output strings, and multiple children are causing me
>>> trouble (entirely too much like real life). Any help or suggestions would
>>> be greatly appreciated. I've created a gist[2] with the following example:
>>>
>>> Thanks very much for your time and trouble.
>>> Best,
>>> Bridger
>>>
>>> ```xquery
>>>
>>> xquery version "3.1";
>>>
>>> declare variable $input :=
>>>   
>>> 
>>>   bbb content
>>> 
>>> 
>>>   bbb content
>>> 
>>> 
>>>   bbb content
>>>   bbb content
>>>   bbb content
>>> 
>>>   ;
>>>
>>> declare function local:elem(
>>>   $nodes as node()*
>>> ) as xs:string* {
>>>   for $node in $nodes
>>>   return(
>>> string-join(
>>>   if ($node/@*)
>>>   then (string-join((name($node) || string-join(for $att in $node/@* 
>>> return local:atty($att))), "/"), local:elem($node/child::*))
>>>   else if ($node/child::*)
>>>then (for $child in $node/child::* return local:elem($child), 
>>> local:elem($node/child::*))
>>>else (name($node) || "/" || local:elem($node/child::*))
>>> )
>>>   )
>>> };
>>>
>>> declare function local:atty(
>>>   $att as attribute()
>>> ) as xs:string* {
>>>   "[@" || name($att) || "='" || data($att) || "']"
>>> };
>>>
>>>
>>> local:e2($input
>>> (:
>>>   this currently returns
>>>   test/aaa/bbb[@type='foo']/
>>>   /
>>>   /aaa/bbb[@type='foo'][@enc='bar']/
>>>   /
>>>   /aaa/bbb[@type='bzz'][@enc='bar']/
>>>   /bbb[@type='qux'][@enc='bar'][@key='yes'][@point='start']/
>>>   /bbb[@type='qux'][@enc='bar'][@key='yes'][@point='end']/
>>>   /
>>>
>>> :)
>>>
>>> ```
>>>
>>> [1] fn:path, and the related functions from functx
>>> (functx:distinct-element-paths, functx:path-to-node, and
>>> functx:path-to-node-with-pos). The functx functions are really close and
>>> awesome, but I need to incorporate attributes into my output.
>>> [2] https://gist.github.com/CanOfBees/8cfb435ac06986c9b0b0c215a786f4d7
>>>
>>>

[basex-talk] Help with a function to return paths

[Bridger Dyson-Smith]

Hi all -

I hope you don't mind a question about serializing distinct XPaths. I'm
trying to generate some reports for documentation and the available
built-in and library functions[1] aren't quite what I need.

The output I'm after is:
test/aaa/bbb[@type="foo"]
test/aaa/bbb[@type="foo"][@enc="bar"]
test/aaa/bbb[@type="bzz"][@enc="bar"]
test/aaa/bbb[@type="qux"][@enc="bar"][@key="yes"][@point="start"]
test/aaa/bbb[@type="qux"][@enc="bar"][@key="yes"][@point="end"]

I have a couple of functions that are getting me close, but I can't quite
manage the output strings, and multiple children are causing me trouble
(entirely too much like real life). Any help or suggestions would be
greatly appreciated. I've created a gist[2] with the following example:

Thanks very much for your time and trouble.
Best,
Bridger

```xquery

xquery version "3.1";

declare variable $input :=
  

  bbb content


  bbb content


  bbb content
  bbb content
  bbb content

  ;

declare function local:elem(
  $nodes as node()*
) as xs:string* {
  for $node in $nodes
  return(
string-join(
  if ($node/@*)
  then (string-join((name($node) || string-join(for $att in
$node/@* return local:atty($att))), "/"), local:elem($node/child::*))
  else if ($node/child::*)
   then (for $child in $node/child::* return
local:elem($child), local:elem($node/child::*))
   else (name($node) || "/" || local:elem($node/child::*))
)
  )
};

declare function local:atty(
  $att as attribute()
) as xs:string* {
  "[@" || name($att) || "='" || data($att) || "']"
};


local:e2($input
(:
  this currently returns
  test/aaa/bbb[@type='foo']/
  /
  /aaa/bbb[@type='foo'][@enc='bar']/
  /
  /aaa/bbb[@type='bzz'][@enc='bar']/
  /bbb[@type='qux'][@enc='bar'][@key='yes'][@point='start']/
  /bbb[@type='qux'][@enc='bar'][@key='yes'][@point='end']/
  /

:)

```

[1] fn:path, and the related functions from functx
(functx:distinct-element-paths, functx:path-to-node, and
functx:path-to-node-with-pos). The functx functions are really close and
awesome, but I need to incorporate attributes into my output.
[2] https://gist.github.com/CanOfBees/8cfb435ac06986c9b0b0c215a786f4d7

Re: [basex-talk] Dpi scaling problem - Ubuntu 20.04

[Bridger Dyson-Smith]

Hi Hugo -

[responding to the list, too, because I'm well-known for missing the
obvious! :)]
On Thu, Oct 15, 2020 at 4:26 PM Silamphre  wrote:

> Hi Bridger,
>
> Thank you for your kind reply and your assistance. I have tried using that
> command directly in the Ubuntu terminal, but unfortunately it works only
> when I set the value to 2, which is way too big for an interface aha. It
> seems to only be working with integers.
> I'll try editing the script, but since it refers to a java program and I'm
> not familiar with it, do you know if I can just copy that command line in
> the script, or if I need to write a more specific command line ?
>
> On a different OS, I had problems with JDK font rendering, so I was adding
flags like this to BaseX in the `basexgui` script like so:

```shell

# Run code
exec java -cp "$CP" -Dsun.java2d.uiScale=2.0 $BASEX_JVM org.basex.BaseXGUI
"$@"
```

i.e. adding the flag *after* the `-cp`/classpath flag in the command.

I just tried this on my current Linux system, and by bumping the number a
bit, I did increase some of the scaling, but only in the editor pane of the
GUI; i.e. I noticed the scaling particularly in the tabs. To be honest, I'm
not sure what to make of that, or how to take a different approach. Maybe
Christian or another use can give us some insights.



> Thank you again.
>
You are most welcome. Sorry that we couldn't get it completely solved.

Kind regards,
> Hugo
>
> Best,
Bridger


> Le jeu. 15 oct. 2020 à 19:32, Bridger Dyson-Smith 
> a écrit :
>
>> Hi Silamphre,
>>
>> I'm not sure if this will help or not, but editing the `basexgui` script
>> to include `-Dsun.java2d.uiScale=1.25` might help[1]? I confess that I
>> don't have UI scaling enabled on my unix-like system, or maybe you've
>> already tried that approach.
>> Hope that helps!
>> Best,
>> Bridger
>>
>> [1]
>> https://stackoverflow.com/questions/58699877/how-to-fix-scaling-of-a-java-based-application-on-a-high-dpi-display-in-linux
>>
>> On Thu, Oct 15, 2020 at 12:06 PM Silamphre 
>> wrote:
>>
>>> Hello everyone,
>>>
>>> I started using BaseX for a class project a few days ago. I'd like to
>>> run it on my Ubuntu 20.04. I've installed openjdk version 14 to do so.
>>>
>>> I've set my Ubuntu display settings at 125% fractional scaling, but it
>>> appears that BaseX does not scale with this setting. The GUI consequently
>>> appears really tiny, almost unusable.
>>>
>>> Does someone have a way to solve this issue, and make BaseX scales
>>> accordingly to my Ubuntu display settings, please?
>>>
>>> Thank you for your help.
>>>
>>> Kind regards,
>>> Hugo
>>>
>>>
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>>>  Garanti
>>> sans virus. www.avast.com
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>>> <#m_8249631326652388190_m_-8493036691201945712_m_240745489779756524_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>

Re: [basex-talk] Dpi scaling problem - Ubuntu 20.04

[Bridger Dyson-Smith]

Hi Silamphre,

I'm not sure if this will help or not, but editing the `basexgui` script to
include `-Dsun.java2d.uiScale=1.25` might help[1]? I confess that I don't
have UI scaling enabled on my unix-like system, or maybe you've already
tried that approach.
Hope that helps!
Best,
Bridger

[1]
https://stackoverflow.com/questions/58699877/how-to-fix-scaling-of-a-java-based-application-on-a-high-dpi-display-in-linux

On Thu, Oct 15, 2020 at 12:06 PM Silamphre 
wrote:

> Hello everyone,
>
> I started using BaseX for a class project a few days ago. I'd like to run
> it on my Ubuntu 20.04. I've installed openjdk version 14 to do so.
>
> I've set my Ubuntu display settings at 125% fractional scaling, but it
> appears that BaseX does not scale with this setting. The GUI consequently
> appears really tiny, almost unusable.
>
> Does someone have a way to solve this issue, and make BaseX scales
> accordingly to my Ubuntu display settings, please?
>
> Thank you for your help.
>
> Kind regards,
> Hugo
>
>
> 
>  Garanti
> sans virus. www.avast.com
> 
> <#m_240745489779756524_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

Re: [basex-talk] XPTY0004 error when calling http:send-request inside of a local function

[Bridger Dyson-Smith]

Christian,

As always, thank you for your help! I had tried the following

```xquery
declare function local:grab2(
  $url as xs:string,
  $path as xs:string,
  $fname as xs:string
) as document-node() {
  let $req := http:send-request(, $url)
  return
if (head($req)/@status = "200")
then file:write($path || $fname, tail($req), map { "method": "xml" })
else document { element error { "something went wrong" } }
};
```
and had the same error - adding an occurrence indicator on the return type
solves it for that trial, too.

Much appreciated!
Best,
Bridger

On Sun, Oct 11, 2020 at 4:06 PM Christian Grün 
wrote:

> Hi Bridger,
>
> As your query will either return an empty sequence (in the case of
> success) or a string (in the case of an error), you’ll need to use
> xs:string? as return type:
>
> declare function local:grab(
>   $url as xs:string,
>   $path as xs:string,
>   $fname as xs:string
> ) as xs:string? {
>   ...
>
> Hope this helps, cheers,
> Christian
>
>
>
> >   let $req := http:send-request(, $url)
> >   return
> > if (head($req)/@status = "200")
> > then file:write($path || $fname, tail($req), map { "method": "xml" })
> > else ("something went wrong!")
> > };
> > ```
> >
> > but I've noticed that in version 9.4.3 this returns an  XPTY error:
> > Stopped at /home/bridger/bin/basex-src/basex-943-xpty0004-error.xq, 1/18:
> > [XPTY0004] Cannot promote xs:string* to document-node(): (let $req_7 :=
> http:send-request(, "
> https://dpla.lib.utk.edu/repox/OAIHa
> >
> > While this could be a bug, should I be writing this kind of local
> function differently? Or could I improve my typing on it?
> >
> > Thanks very much for any advice or insights you can provide.
> > Best,
> > Bridger
>

Re: [basex-talk] XPTY0004 error when calling http:send-request inside of a local function

[Bridger Dyson-Smith]

And naturally I forgot to mention...
On Sun, Oct 11, 2020 at 3:49 PM Bridger Dyson-Smith 
wrote:

> Hi all -
>
> I'm accustomed to writing something like the following:
>
> ```xquery
>
> declare function local:grab(
>   $url as xs:string,
>   $path as xs:string,
>   $fname as xs:string
> ) as document-node() {
>   let $req := http:send-request(, $url)
>   return
> if (head($req)/@status = "200")
> then file:write($path || $fname, tail($req), map { "method": "xml" })
> else ("something went wrong!")
> };
> ```
>
> but I've noticed that in version 9.4.3 this returns an  XPTY error:
> Stopped at /home/bridger/bin/basex-src/basex-943-xpty0004-error.xq, 1/18:
> [XPTY0004] Cannot promote xs:string* to document-node(): (let $req_7 :=
> http:send-request(, "
> https://dpla.lib.utk.edu/repox/OAIHa
>
> that despite the error, the correct response is written to the filepath
specified.

While this could be a bug, should I be writing this kind of local function
> differently? Or could I improve my typing on it?
>
> Thanks very much for any advice or insights you can provide.
>
Apologies for the omission!


> Best,
> Bridger
>

[basex-talk] XPTY0004 error when calling http:send-request inside of a local function

[Bridger Dyson-Smith]

Hi all -

I'm accustomed to writing something like the following:

```xquery

declare function local:grab(
  $url as xs:string,
  $path as xs:string,
  $fname as xs:string
) as document-node() {
  let $req := http:send-request(, $url)
  return
if (head($req)/@status = "200")
then file:write($path || $fname, tail($req), map { "method": "xml" })
else ("something went wrong!")
};
```

but I've noticed that in version 9.4.3 this returns an  XPTY error:
Stopped at /home/bridger/bin/basex-src/basex-943-xpty0004-error.xq, 1/18:
[XPTY0004] Cannot promote xs:string* to document-node(): (let $req_7 :=
http:send-request(, "
https://dpla.lib.utk.edu/repox/OAIHa

While this could be a bug, should I be writing this kind of local function
differently? Or could I improve my typing on it?

Thanks very much for any advice or insights you can provide.
Best,
Bridger

Re: [basex-talk] improving query performance

[Bridger Dyson-Smith]

Hi Bill,

I won't claim to be any kind of query expert, but there are some things you
can try to experiment with query speeds:

On Fri, Aug 21, 2020 at 3:52 PM Bill Osmond  wrote:

> Hi all,
> I'm attempting to query a fairly large database, with 136,938 resources
> and a size of 12,257,686,099. The basex server itself is an AWS EC2
> instance with 4 cores & 16 gigs of ram, using -Xmx12g. The database
> contains audio DDEX information which (to me at least) is fairly
> complicated XML. The query I'm running against the database works and gives
> me the result I want, but takes upwards of 15 minutes to execute. I've
> taken a look at the wiki documentation regarding indexes, but it's not
> clear to me that any of the non-default indexes would help for the query
> I'm running. That query is:
>
> declare namespace ernm="http://ddex.net/xml/ern/411";;
>
> for $r in /ernm:NewReleaseMessage
> for $track_release in $r/ReleaseList/TrackRelease
> for $party in $r/PartyList/Party
> for $sound_recording in $r/ResourceList/SoundRecording
> for $release in $r/ReleaseList/Release
> where
>   $track_release/ReleaseLabelReference = $party/PartyReference
>   and $track_release/ReleaseResourceReference =
> $sound_recording/ResourceReference
>   and $track_release/ReleaseResourceReference =
> $release/ResourceGroup/ResourceGroup/ResourceGroupContentItem/ReleaseResourceReference
> return
>   
> { $track_release/ReleaseId/ISRC/text() }
> { fn:string-join($sound_recording/DisplayArtistName, '/')
> }
> { $sound_recording/DisplayTitleText/text() }
> { $release/DisplayTitleText/text() }
> { $release/ReleaseId/ICPN/text() }
> { $party/PartyName/FullName/text() }
>   
>
> Am I wrong, and would an additional value index help here? Or is my query
> just bad?
>
> It looks like you're trying to use text values -- do you have a full range
of indexes in your database? (text, attribute, token, and maybe full text)

Also, have you tried running this query against a local instance of the
database? The GUI can help steer you towards query optimizations in the
Info window.


> Thanks,
> Bill
>
> p.s. The BaseX GUI client is awesome - especially the treemap view. Really
> helps with wrangling these XML files!
>

I'm crunched for time at the moment so please excuse the hasty response.

Best,
Bridger

Re: [basex-talk] BaseX 9.4: The Summer Edition

[Bridger Dyson-Smith]

Christian and the rest of the BaseX team -

As always, thanks for all of your efforts - this release looks very
exciting!
Best,
Bridger

On Tue, Jul 14, 2020 at 7:40 AM Christian Grün 
wrote:

> We are glad to give you Version 9.4 of BaseX, our XML framework,
> database system and XQuery 3.1 processor:
>
>   https://basex.org/
>
> We have focused on rewriting and optimizing complex XQuery code and
> speeding up your RESTXQ apps:
>
> WEB APPLICATIONS
> - DBA: support for millions of log entries
> - LOGTRACE option: write trace output to logs or STDERR
> - rest:init: Full or partial initialization of RESTXQ cache
> - basexhttp: register job services
>
> GUI
> - result view: show number of results and result size
> - Shift F4-F7: toggle search options (case, regeex, ..)
> - Ctrl-Shift-B: jump to matching bracket
>
> XQUERY LOCKING
> - single lock option for reading and updating queries
> - Java bindings: annotation added for updating functions
>
> XQUERY FUNCTIONS
> - convenience functions: json:doc, csv:doc, html:doc
> - element names: convert:encode-key, convert:decode-key
> - jobs:eval: option added for writing log entries
> - session module: only create new sessions if required
>
> XQUERY PERFORMANCE
> - boolean comparisons: flatten nested expressions
> - boolean expressions: merge conjunctions
> - comparisons: simplify operands with if expression
> - database functions: always open at compile time
> - databases: propagate to more expressions at compile time
> - filters: inline context for single items
> - filters: rewrite to simple maps
> - FLWOR: inline where clauses into let clauses
> - FLWOR: merge last with return clause, rewrite to simple map
> - FLWOR: rewrite "return if ..." to "where ... return"
> - fn:for-each, fn:filter: rewrite to FLWORs and filters
> - if expression, EBV tests: simplify boolean tests
> - lists, sets, logical expressions: flatten expressions
> - lists: rewrite to union expressions
> - logical expressions: apply more boolean algebra rules
> - paths: remove redundant predicates
> - predicates with name functions: rewrite to name tests
> - predicates: merge expressions, discard redundant tests
> - predicates: remove of superfluous and redundant tests
> - set expressions: merge operands
> - simple map: inline cast expressions
> - simple maps: faster evaluation
> - simple maps: merge operands
> - simple maps: rewrite to paths, optimize for index access
> - switch: rewrite to if expression
> - types: skip redundant checks, promotions  and conversions
>
> For a more comprehensive list of added and updated features, check out
> our documentation (docs.basex.org) and check out the GitHub issues
> (github.com/BaseXdb/basex/issues).
>
> Have fun,
> Your BaseX Team
>

Re: [basex-talk] [FULLTEXT] How to search for underscore seperated words ?

[Bridger Dyson-Smith]

Fabrice - my apologies. I see now that my suggestion was very naive. I hope
I didn't waste your time.
Sorry for the noise.
Best,
Bridger

On Fri, Jun 5, 2020 at 10:23 AM Bridger Dyson-Smith 
wrote:

> Hi Fabrice -
>
> Maybe something like
> ft:search("the-database", "YET_ANOTHER_SILLY_KEYWORD", map { "mode":
> "phrase", "ordered": true() })?
>
> I confess that I don't know the full text module behavior at all, but I
> coincidentally had it open... Anyway, I hope that's helpful.
> Best,
> Bridger
>
> On Fri, Jun 5, 2020 at 8:38 AM ETANCHAUD Fabrice <
> fabrice.etanch...@maif.fr> wrote:
>
>> Hi all BaseX users,
>>
>> When I search for 'YET_ANOTHER_SILLY_KEYWORD', ft:search gives me all
>> text nodes containing any of the YET ANOTHER SILLY KEYWORD words.
>> I tried the different search modes, but could not find a way to get
>> ft:search return only the text nodes containing the concatenation.
>> Does anybody have a solution ?
>>
>> Best regards,
>>
>>
>> [image: MAIF]
>>
>> *Fabrice ETANCHAUD*
>>
>> Unité Etudes Informatiques - MAIF VIE
>>
>> 50 avenue Salvador Allende 79029 Niort Cedex 9
>>
>> fabrice.etanch...@maif.fr
>>
>> *MAIF*.FR <http://www.maif.fr/> | Retrouvez-nous aussi sur [image:
>> Facebook] <https://www.facebook.com/MAIFassureur>[image: Twitter]
>> <https://twitter.com/MAIF>[image: Youtube]
>> <https://www.youtube.com/user/maif>
>>
>>
>>

Re: [basex-talk] [FULLTEXT] How to search for underscore seperated words ?

[Bridger Dyson-Smith]

Hi Fabrice -

Maybe something like
ft:search("the-database", "YET_ANOTHER_SILLY_KEYWORD", map { "mode":
"phrase", "ordered": true() })?

I confess that I don't know the full text module behavior at all, but I
coincidentally had it open... Anyway, I hope that's helpful.
Best,
Bridger

On Fri, Jun 5, 2020 at 8:38 AM ETANCHAUD Fabrice 
wrote:

> Hi all BaseX users,
>
> When I search for 'YET_ANOTHER_SILLY_KEYWORD', ft:search gives me all text
> nodes containing any of the YET ANOTHER SILLY KEYWORD words.
> I tried the different search modes, but could not find a way to get
> ft:search return only the text nodes containing the concatenation.
> Does anybody have a solution ?
>
> Best regards,
>
>
> [image: MAIF]
>
> *Fabrice ETANCHAUD*
>
> Unité Etudes Informatiques - MAIF VIE
>
> 50 avenue Salvador Allende 79029 Niort Cedex 9
>
> fabrice.etanch...@maif.fr
>
> *MAIF*.FR  | Retrouvez-nous aussi sur [image:
> Facebook] [image: Twitter]
> [image: Youtube]
> 
>
>
>

Re: [basex-talk] Help composing an http:request with POST

[Bridger Dyson-Smith]

Hi all -

Well... after staggering around and clutching my head for a few hours I
realized that I didn't need the `multipart` at all (tricked by tricky API
documentation, once again!).

http:send-request(

  
  
,
'http
://localhost:8080/fedora/objects/tdh:2/datastreams/OBJ?versionable=true?dsLocation=/path/to/my/
image.tif',
file:read-binary('/path/to/my/image.tif')
)

is working great.

Have a lovely day and/or night.
Best,
Bridger


On Wed, May 20, 2020 at 5:40 PM Bridger Dyson-Smith 
wrote:

> Hi all -
>
> I'm trying to use BaseX to POST an image to a digital asset management
> system (i.e. I'm not using RESTXQ, etc, just using BaseX to compose and
> send the request).
>
> The request:
> http:send-request(
>  auth-method='Basic' send-authorization='true'>
>   
>   
>   
> ,
> '
> http://localhost:8080/fedora/objects/tdh:2/datastreams/OBJ?versionable=true?dsLocation=/path/to/my/image.tif
> ',
> file:read-binary('/path/to/my/image.tif')
> )
>
> I'm a hapless caveman when it comes to http, and so when the binary
> content gets POSTed, the mime-type becomes something like
> 'multipart/form-data;boundary=1BEF0A57BE110FD467A'. I'm clearly missing
> something in the http:request but... I don't know what.
>
> Any suggestions? It seems like I need some training wheels here, but I'm
> not sure which ones. :)
> Thank you, in advance, for your time and trouble.
> Best,
> Bridger
>

[basex-talk] Help composing an http:request with POST

[Bridger Dyson-Smith]

Hi all -

I'm trying to use BaseX to POST an image to a digital asset management
system (i.e. I'm not using RESTXQ, etc, just using BaseX to compose and
send the request).

The request:
http:send-request(

  
  
  
,
'
http://localhost:8080/fedora/objects/tdh:2/datastreams/OBJ?versionable=true?dsLocation=/path/to/my/image.tif
',
file:read-binary('/path/to/my/image.tif')
)

I'm a hapless caveman when it comes to http, and so when the binary content
gets POSTed, the mime-type becomes something like
'multipart/form-data;boundary=1BEF0A57BE110FD467A'. I'm clearly missing
something in the http:request but... I don't know what.

Any suggestions? It seems like I need some training wheels here, but I'm
not sure which ones. :)
Thank you, in advance, for your time and trouble.
Best,
Bridger

Re: [basex-talk] BaseX 9.3.3: Query optimizations, Bug Fixes

[Bridger Dyson-Smith]

Christian, et al,
As always thank you, and everyone on the BaseX team, for all of your
efforts and hard work!

Best,
Bridger

On Fri, May 15, 2020 at 5:35 AM Christian Grün 
wrote:

> Dear users and supporters of BaseX,
>
> We have just released a new maintenance version of our XML framework,
> database and XQuery processor:
>
>http://basex.org
>
> Apart from minor bug fixes, and in preparation to BaseX 9.4, we have
> added even more query optimizations, FLWOR, path and predicate
> rewritings. Feel free to check out the full list of added rewritings
> and enhancements on Github:
>
>   https://github.com/BaseXdb/basex/issues?q=is%3Aissue+is%3Aclosed
>
> Have fun with the new release!
>
> Christian
> BaseX Team
>
> PS: With BaseX 10 (no release date yet), we’ll switch from Java 8 to
> 11, the current long-term-support release of Java.
>

Re: [basex-talk] Unconfirmed bug: NPE using cached transformer

[Bridger Dyson-Smith]

Hi Andrew and list,

I ran into a similar error with 9.3.3-beta this last week but haven't had a
chance to work up a SSCCE.  Your report makes me think I need to do so!

Best,
Bridger

On Sat, Apr 25, 2020, 11:19 AM Andrew Sales  wrote:

> Hello,
>
> I'd like to run a transform on multiple documents using a cached
> transformer, accumulating any errors that occur.
>
> This minimal example -
>
> for $doc in (,, )
> return
> try{
> xslt:transform(
>   $doc,
>   http://www.w3.org/1999/XSL/Transform";
> version='2.0'>
> 
>   ,
>   map{},
>   map{'cache':'true'}
> )
> }
> catch * {$err:description}
>
> - results in:
>
> Error in xsl:sequence/@select on line 1 column 135 of [...]:
>   FOER: Error signalled by application call on error()Improper use?
> Potentia
> l bug? Your feedback is welcome:
> Contact: basex-talk@mailman.uni-konstanz.de
> Version: BaseX 9.3.2
> Java: Oracle Corporation, 1.8.0_241
> OS: Windows 7, amd64
> Stack Trace:
> java.lang.NullPointerException
> at org.basex.io.out.ArrayOutput.write(ArrayOutput.java:32)
> at java.io.OutputStream.write(Unknown Source)
> at java.io.PrintStream.write(Unknown Source)
> at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
> at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
> at sun.nio.cs.StreamEncoder.flushBuffer(Unknown Source)
> at java.io.OutputStreamWriter.flushBuffer(Unknown Source)
> at java.io.PrintStream.write(Unknown Source)
> at java.io.PrintStream.print(Unknown Source)
> at java.io.PrintStream.println(Unknown Source)
> at net.sf.saxon.lib.StandardLogger.println(StandardLogger.java:121)
> at net.sf.saxon.lib.Logger.error(Logger.java:45)
> at
> net.sf.saxon.lib.StandardErrorListener.fatalError(StandardErrorListen
> er.java:305)
> at net.sf.saxon.Controller.reportFatalError(Controller.java:958)
> at net.sf.saxon.Controller.transform(Controller.java:1914)
> at
> net.sf.saxon.s9api.XsltTransformer.transform(XsltTransformer.java:579
> )
> at
> net.sf.saxon.jaxp.TransformerImpl.transform(TransformerImpl.java:185)
>
> at
> org.basex.query.func.xslt.XsltTransform.transform(XsltTransform.java:
> 135)
> at
> org.basex.query.func.xslt.XsltTransform.transform(XsltTransform.java:
> 62)
> at
> org.basex.query.func.xslt.XsltTransform.item(XsltTransform.java:40)
> at org.basex.query.expr.ParseExpr.value(ParseExpr.java:51)
> at org.basex.query.expr.Try.value(Try.java:94)
> at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46)
> at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:72)
> at org.basex.query.QueryContext.next(QueryContext.java:353)
> at org.basex.query.scope.MainModule$1.next(MainModule.java:104)
> at org.basex.core.cmd.AQuery.query(AQuery.java:109)
> at org.basex.core.cmd.XQuery.run(XQuery.java:22)
> at org.basex.core.Command.run(Command.java:257)
> at org.basex.core.Command.execute(Command.java:93)
> at org.basex.api.client.LocalSession.execute(LocalSession.java:132)
> at org.basex.api.client.Session.execute(Session.java:36)
> at org.basex.core.CLI.execute(CLI.java:92)
> at org.basex.BaseX.(BaseX.java:107)
> at org.basex.BaseX.main(BaseX.java:43)
>
> The exception does not occur if 'cache' is set to 'false'.
>
> Relatedly, if I amend the query to this:
>
> for $doc in (,,)
> return
> try{
> xslt:transform(
>   $doc,
>   http://www.w3.org/1999/XSL/Transform";
>   xmlns:xs="http://www.w3.org/2001/XMLSchema"; version='2.0'>
> 
>   
> 
>   ,
>   map{},
>   map{'cache':'true'}
> )
> }
> catch * {{$err:description}}
>
> the error message is passed through for the first item in the sequence
> (), but is lost,  resulting in  when adjusted to error on
> subsequent items ( or ).
>
> Again, this works as expected if the transformer is not cached.
>
> Thanks and regards,
> Andrew
>
>
> 
>  Virus-free.
> www.avast.com
> 
> <#m_-7045194468050368418_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

Re: [basex-talk] How to apply array:for-each on an array of arrays?

[Bridger Dyson-Smith]

Hi Ben -
I'm on mobile, please excuse any typos.

Maybe
`return array { $idf }`
is closer?

Untested, apologies!
Best,
Bridger


On Mon, Mar 30, 2020, 5:16 PM Ben Engbers  wrote:

> Hi,
>
> In textmining, the 'idf' or inverse document frequency is defined as
> idf(term)=ln(ndocuments / ndocuments containing term). I am working on a
> function that should return this idf.
>
> This function:
>
> declare function local:wordFreq_idf($nodes as node()*) as array(*) {
>   let $count := count($nodes)
>   let $text := for $node in $nodes
>  return $node/text() => tokenize() => distinct-values()
>  let $idf := $text   => tidyTM:wordCount_arr()
>   return $idf
> };
>
> returns:
>
> ["probleem", 703]
> ["opgelost.", 248]
> ["dictu", 235]
> ["opgelost", 217]
> ["medewerker", 193]
> ...
>
> For "probleem", the idf should be calculated as ln($count/703). Since
> there are 1780 nodes this would result in 0.929011751.
> I tried to exten the 'let $idf' line with:
>=> array:for-each(function($idf) {array:append($idf,
> math:log($count div $idf[2]) )})
> which should result in ["probleem", 703, 0.929011751]
>
> but no mather what I do, every time I get this error:
> [XPTY0004] Cannot promote (array(xs:anyAtomicType))+ to array(*): ([
> "probleem", 703 ], [ "opgelost.", 248 ], ...).
>
> Is it possible to apply array:for-each on an array of arrays?
>
> Ben
>
>

Re: [basex-talk] BaseX 9.3.2: Performance Tweaks, Bug Fixes

[Bridger Dyson-Smith]

Christian -

echoing Marco, indeed what happy news! Thank you and the rest of the BaseX
team for all of your efforts!

Best,
Bridger

On Tue, Mar 10, 2020 at 9:19 AM Christian Grün 
wrote:

> Dear all,
>
> We are glad to provide you with a new BaseX 9.3.2 maintenance release:
>
>http://basex.org
>
> In preparation to version 9.4, we have introduced numerous small query
> optimizations that speed up path, predicate and simple map operations
> and will rewrite your queries for index access more often than before.
> Feel free to check out the full list of added rewritings and
> enhancements on Github:
>
>   https://github.com/BaseXdb/basex/issues?q=is%3Aissue+is%3Aclosed
>
> Have fun with the new release,
>
> Christian
> BaseX Team
>

Re: [basex-talk] increment a variable only when a conditional is true?

[Bridger Dyson-Smith]

Hi Thufir -

Maybe something like this will help?

```
xquery version "3.1";

let $y := 99

for $x in (1 to 9)
count $iterator
let $decrease := $y - $iterator
return(
  comment { "iterator = " ||  $iterator },
  comment { "decrease = " || $decrease },
  
)
```

I'm basically ripping Walmsley's book off for this example -- see pages
~135-7 (examples P-6,7). The `count` clause makes this work.

Best,
Bridger

PS Remember, the first step in avoiding a *trap* is knowing of its
existence. :)

On Wed, Feb 19, 2020 at 10:33 AM thufir  wrote:

> How do I decrement y?
>
> Pardon, output for a simpler example:
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> the FLWOR:
>
> xquery version "3.0";
> let $y := 99
> for $x in (1 to 9)
>   let $y := $y - 1
> return 
>
>
> is it not possible to decrement $y without using some external scripting
> function?  That seems odd.
>
>
> thanks,
>
> Thufir
>

Re: [basex-talk] Flat XML fods database

[Bridger Dyson-Smith]

Hi Thufir,

You might find Priscilla Walmsley's Functx library[1] section on namespaces
helpful (or something else interesting in the functions listed there).

Michael - that's a very nice FLOWR! Thank you for sharing.

Best,
Bridger

[1] http://www.xqueryfunctions.com/xq/c0021.html

On Fri, Feb 14, 2020, 7:29 AM Michael Seiferle  wrote:

> Hi Thufir,
>
> I think you might look up the namespaces from the Libre Office File like
> so:
>
> (
>   for $node in $doc//(@*,*)
>   let $nn := node-name($node)
>   let $ns-uri := namespace-uri-from-QName($nn)
>   let $prefix := prefix-from-QName($nn)
>   where $ns-uri
>   return ``[declare namespace `{ $prefix }` = "`{ $ns-uri }`"]``
> ) => distinct-values()
> => string-join(out:nl())
>
> https://git.basex.io/snippets/77
>
>
> Other than that you might have to consult the Schema files I guess :)
>
>
> Am 14.02.2020 um 09:27 schrieb thufir :
>
> I think I mainly need to add a namespace for fods:
>
>
> thufir@dur:~/fods/flwor$
> thufir@dur:~/fods/flwor$ basex text.xq
> Stopped at /home/thufir/fods/flwor/text.xq, 3/14:
> [XPST0081] No namespace declared for 'text:p'.
> thufir@dur:~/fods/flwor$
> thufir@dur:~/fods/flwor$ cat text.xq
>
> for $foo  in db:open("foo")
> return $foo//text:p
>
> thufir@dur:~/fods/flwor$
>
>
> https://stackoverflow.com/q/60222478/262852
>
>
>
> how do I know all the namespacess?  from the libre office file?
>
>
>

Re: [basex-talk] filtering NaN from a sequence

[Bridger Dyson-Smith]

Hi Graydon,
I'm mobile at the moment, so please excuse the abbreviated reply. Would
functx:is-a-number() [#1] work in your where clause?

I'm completely unable to test... apologies.

Best,
Bridger

#1 http://www.xqueryfunctions.com/xq/functx_is-a-number.html

On Sun, Feb 2, 2020, 7:22 PM Graydon Saunders  wrote:

> Hello all --
>
> So I have a CSV file, and I can pull that into BaseX in the hopes of
> writing a query to extract a report.  I'm using 9.3.1 for the purpose.
>
> Not all of the Payment_Amount fields have a value, so any
> report-extracting query has to filter those out of any calculations or the
> whole thing gets infested with NaN.
>
> This works:
> let $xmlReport as document-node(element(csv)) :=
>  file:read-text('report.csv') => csv:parse( map { 'header': true(),
> 'separator' : 'tab' })
>
> let $made as xs:double+ := for $value in
> $xmlReport/csv/record/Payment_Amount[text() castable as xs:double]/number()
>   return $value
>
> return sum($made) => round(2)
>
> If I wanted to use a where clause,
>
> let $xmlReport as document-node(element(csv)) :=
>  file:read-text('report.csv') => csv:parse( map { 'header': true(),
> 'separator' : 'tab' })
>
> let $made as xs:double+ := for $value in
> $xmlReport/csv/record/Payment_Amount/number()
>   where ???
>   return $value
>
> return sum($made) => round(2)
>
> What do I put in the where clause?  I tried
> where not($value = NaN)
> and that was not successful:
> "Stopped at /home/graydon/git/writing/transform/urk.xq, 6/25:
> [XPTY0020] element(NaN): node expected, xs:double found: 3.38."
>
> where not($value = number('NaN'))
>
> didn't give an error but the query returns NaN so I know I didn't filter
> any of the empty records from the sum.
>
> How ought that where clause be written?
>
> Thanks!
> Graydon
>
>

Re: [basex-talk] Bug merging maps with option duplicates:combine?

[Bridger Dyson-Smith]

Hi all,

While I don't have any answer to this question, I can corroborate Johannes'
finding.

Also, Johannes, did you find a magic string? I tried some longer keys and
they seem to work :)
Best,
Bridger

On Thu, Jan 9, 2020 at 11:51 AM Johannes Echterhoff <
echterh...@interactive-instruments.de> wrote:

> Hello,
>
> When executing:
>
>
>
> map:merge((
> map {
> "DENW21AL100077Hs": "a",
> "DENW21AL100076i5": "a"
> },
> map {
> "DENW21AL100077Hs": "b",
> "DENW21AL100076i5": "b",
> "DENW21AL100076hT": "b"
> }
> ), map {'duplicates': 'combine'})
>
>
>
> in the BaseX GUI I get:
>
>
>
> Error:
>
> Improper use? Potential bug? Your feedback is welcome:
>
> Contact: basex-talk@mailman.uni-konstanz.de
>
> Version: BaseX 9.3.1
>
> Java: AdoptOpenJDK, 1.8.0_212
>
> OS: Windows 10, amd64
>
> Stack Trace:
>
> java.lang.ArrayIndexOutOfBoundsException: 2
>
> at
> org.basex.query.value.map.TrieList.add(TrieList.java:199)
>
> at
> org.basex.query.value.map.TrieList.addAll(TrieList.java:141)
>
> at org.basex.query.value.map.XQMap.addAll(XQMap.java:106)
>
> at org.basex.query.func.map.MapMerge.item(MapMerge.java:33)
>
> at
> org.basex.query.func.StandardFunc.optimize(StandardFunc.java:82)
>
> at org.basex.query.expr.Arr.compile(Arr.java:43)
>
> at
> org.basex.query.scope.MainModule.comp(MainModule.java:81)
>
> at
> org.basex.query.QueryCompiler.compile(QueryCompiler.java:114)
>
> at
> org.basex.query.QueryCompiler.compile(QueryCompiler.java:105)
>
> at
> org.basex.query.QueryContext.compile(QueryContext.java:312)
>
> at
> org.basex.query.QueryProcessor.compile(QueryProcessor.java:79)
>
> at org.basex.core.cmd.AQuery.query(AQuery.java:93)
>
> at org.basex.core.cmd.XQuery.run(XQuery.java:22)
>
> at org.basex.core.Command.run(Command.java:257)
>
> at org.basex.core.Command.execute(Command.java:93)
>
> at org.basex.gui.GUI.exec(GUI.java:422)
>
> at org.basex.gui.GUI.lambda$4(GUI.java:365)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Best regards,
>
> Johannes
>


map-merge-bug.xq
Description: Binary data

Re: [basex-talk] BaseX 9.3: The Winter Edition

[Bridger Dyson-Smith]

Christian, and the BaseX Team -

Congratulations on the new release!
Thank you for all of your time and efforts.

Best,
Bridger

On Fri, Nov 29, 2019 at 11:14 AM Christian Grün  wrote:

> Hello to our loyal readers of our list,
>
> We are glad to announce version 9.3 of BaseX, our XML framework,
> database system and XQuery 3.1 processor:
>
>   http://basex.org/
>
> The focus has been put on in-depth optimizations of complex XQuery
> code and faster processing of RESTXQ web applications:
>
> RESTXQ
> - seamless streaming of custom responses to the client
> - Request Module: bind attributes to current request
> - better support for the OPTIONS and HEAD methods
> - trace output is now stored in the database logs
> - enforce premature HTTP response with web:error
> - optional GZIP compression of HTTP responses
> - server-side forwarding with web:forward
>
> USER INTERFACES
> - GUI: project view: skip directories with .ignore file
> - GUI: project view: do now follow symbolic links
> - DBA: user panel updated to show user info
> - DBA: faster processing of log entries
>
> STORAGE
> - hot fix for replacing single documents with namespaces
> - improved XQuery processing of binary resources
> - ADDCACHE: faster caching of single documents
> - WITHDB: less locking when using fn:doc
>
> XQUERY FUNCTIONS
> - user:info/user:update-info: process user-specific data
> - db:open-id, db:open-pre: support for multiple ids
> - file:descendants: return descendant file paths
> - inspect:type: return type of a value as string
> - db:alter-backup: rename database backups
> - xslt:transform: support for catalog files
>
> XQUERY PERFORMANCE
> - merge of adjacent/redundant steps in paths and predicates
> - removal of redundant switch, typeswitch and catch clauses
> - interruption of side-effecting code (Files Modul etc.)
> - fn:count: pre-evaluation of zero-or-one arguments
> - faster processing of alternative steps: a/(b,c)
> - rewriting of paths to simple map expressions
> - inlining of values in simple map expressions
> - flattening of nested simple map expressions
> - rewriting of lists to union expressions
> - removal of superfluous predicates
>
> For a more comprehensive list of added and updated features. please
> check out our documentation: http://docs.basex.org/.
>
> Your feedback is welcome as usual.
>
> Have fun with the new release,
> Your BaseX Team
>

Re: [basex-talk] http:request parsing question

[Bridger Dyson-Smith]

Hi Christian,
as usual, you're very helpful!

One more example of how it can really help to pay attention to those
function signatures -- I wasn't thinking about the http:response as a
sequence, so thank you very much for that insight!

Best,
Bridger

On Mon, Oct 28, 2019 at 10:13 PM Christian Grün 
wrote:

> Hi Bridger,
>
> > http:send-request()/h:response/@status
> > fails.
>
> The reason is that http:send-request returns an http:response element
> as first item. "/h:response" is just another writing for
> "/child::h:response", As the response has no other response child, you
> won’t get any result. This is what you can do instead:
>
> 1. Use the self axis:
>
>   let $data := http:send-request( href='http://export.arxiv.org/oai2?verb=Identify'/>)
>   return $data/self::http:response/@status
>
> 2. Return the status attribute from the first result (which will
> always be the http:response element):
>
>   let $data := http:send-request( href='http://export.arxiv.org/oai2?verb=Identify'/>)
>   return head($data)/@status
>
> Hope this helps,
> Christian
>

Re: [basex-talk] Understanding the path index

[Bridger Dyson-Smith]

Julia -
Wow, thank you so much for your detailed response -- your project sounds
really neat!

I'm sorry to say that I forgot to speak to your question:

On Mon, Oct 28, 2019 at 5:51 AM Beck, Julia 
wrote:

> Hi Bridger,
>
> thanks for your thoughts! It might be a good idea to change the
> modeling there. I already have extra databases for other entities
> (agents, places, events, ...), so it might be a good idea to separate
> the web resources from the rest, too. But I have to test it. I have not
> done it yet, because I tried to follow the EDM structure where the web
> resource is one of the core classes in the model, so I tried to keep it
> together with the aggregation and the acutal object. Thanks for making
> me rethink this.
>


> Though I am still curious about why this performance difference occurs.
>
> I think (but am not sure) that the performance hit is due to BaseX having
to examine all 5000 webResources sequentially to compare the
webResource[@rdf:about] to the $urn value (note: I'm probably saying
something egregiously incorrect in that sentence: hopefully Christian, or
another knowledgeable user, will be along to help us understand things a
bit better).

Does it help at all to wrap the entire query in the pragma?


> For your other questions: We are implementing a search portal with
> metadata from the performing arts domain and for that we are
> aggregating metadata from currently about 20 different data providers
> (museums, libraries, archives). The metadata comes to us in different
> ways of delivery/harvesting and has very different formats and
> standards (none of them is RDF from the beginning). That being said, I
> use BaseX to do manipulation like transforming and enrichment on the
> metadata.
> Outside of BaseX, I am mostly using Python and I ingest the data with
> the Python client to BaseX. Like you, I had out of memory issuses in
> the beginning. But I started to batch all the records to reasonable
> sized packages of about 1000 records (combine files with only one
> record per file to bigger files; split files with e.g. 10 records
> to smaller files). With these batches I have no problem to get the data
> into BaseX and work with it there and outside of BaseX. Even if the
> input is JSON. I also think it is important to use the ADDCACHE option
> when ingesting larger files.
> I have one database for each data provider (actually my original
> question refers to two of those databases [they are not called abc and
> def ;-)], though I left out the object part as it is not relevant
> here). And several databases for the different entities as mentioned
> above. So far, I did not have bigger problems with the namespaces,
> maybe because I am not doing anything specific with them and mostly
> keep them they way they are.
>
> Best wishes,
> Julia
>
>
Kind regards,
Bridger


> On Fri, 2019-10-25 at 15:44 -0400, Bridger Dyson-Smith wrote:
> > Julia, all,
> >
> > apologies - I hit 'send' a bit too quickly. I wanted to ask how you
> > were
> > ingesting RDF, what the size of your input looked like, and how you
> > were
> > dealing with namespaces. I have some RDF/MADS data from the Library
> > of
> > Congress and I've never been able to ingest it into BaseX, even
> > stripping
> > namespaces, etc, things seem to run out of memory or stall.
> >
> > Would you be willing to share some details about your data?
> >
> > Thanks very much.
> > Best,
> > Bridger
> >
> > On Fri, Oct 25, 2019 at 3:41 PM Bridger Dyson-Smith <
> > bdysonsm...@gmail.com>
> > wrote:
> >
> > > Hi Julia -
> > >
> > > Preface: let me be clear when I say that I've wondered about some
> > > of this
> > > myself, so I don't think I have an answer for you. That being said,
> > > I
> > > wonder if this is a grouping/data modeling problem: i.e. you have
> > > 5,000
> > > aggregations refer to 1 of 3 web resources vs ~7,000 aggregations
> > > each with
> > > their own distinct web resource.
> > >
> > > If you created 3 databases for "abc" ( hm... "a", "b", and "c"? ☺),
> > > one
> > > for each web resource (i.e. where there would be only
> > > aggregations-to-the-specific-web-resource), would that help with
> > > query
> > > times at all? It might necessitate a bit of pre-processing in your
> > > creation
> > > step though.
> > >
> > > In any event, I hope those random thoughts are helpful in some way.
> > > Best,
> > > Bridger
> > >
> > >
> > >

Re: [basex-talk] http:request parsing question

[Bridger Dyson-Smith]

Hi all -

On Tue, Oct 22, 2019 at 11:13 PM Bridger Dyson-Smith 
wrote:

> Hi Liam!
>
> On Tue, Oct 22, 2019 at 7:38 PM Liam R. E. Quin 
> wrote:
>
>> On Tue, 2019-10-22 at 19:32 -0400, Bridger Dyson-Smith wrote:
>> >
>> > http://export.arxiv.org/oai2?verb=Identify'/>)//@status/data()
>> >
>> > returns '200', but trying
>> >
>> > http:send-request()/h:response/@status
>> >
>> > fails.
>>
>> I'm guessing that the first request sends you an auth token, and that
>> you're supposed to send this as a parameter in the second token?
>>
>> How exactly does the second call fail?
>>
>> Apologies for not clarifying. The @href points to an OAI-PMH endpoint,
> which simply returns a XML response[1] saying, "This is an OAI-PMH
> endpoint, here's an administrator's email address, plus some other stuff."
> So, there aren't any tokens being passed back in forth in this case (to be
> honest, there are resumption tokens that can be passed back in to an OAI
> provider for other types of requests).
>
> I had thought it might be a good idea to try to think about the HTTP
> status ('200', '404', etc) in a thing I'm working on, and while I realize I
> can get to it with a `//@status` , I had thought a more direct expression
> would be more suited (what if there are other `@status` attributes, etc). I
> even tried taking my shoes off, to no avail. :)
>
> In any case, thank you!
> Best,
> Bridger
>
> After thinking about this some more, I realized that wrapping the overall
HTTP response in my own element let's me write the sorts of expressions I
was thinking about. E.g.
declare namespace http = "http://expath.org/ns/http-client";;
let $r :=
  
{ http:send-request(, '
http://export.arxiv.org/oai2?verb=Identify') }
  
return $r/http:response/@status/data()

I'm curious, though, what other people are doing in the context of their
RESTXQ applications? Are you wrapping responses in elements, or processing
them some other way (I looked through the Web Module[2] documentation but
didn't see anything that looked apropos.

In any case, enjoy your weekend.
Best,
Bridger


>> --
>> Liam Quin, https://www.delightfulcomputing.com/
>> Available for XML/Document/Information Architecture/XSLT/
>> XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
>> Barefoot Web-slave, antique illustrations:  http://www.fromoldbooks.org
>>
>> [1] The full response from arXiv's OAI endpoint looks like this:
> ```
>
>  xmlns="http://www.openarchives.org/OAI/2.0/"; 
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
> xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ 
> http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd";>2019-10-22T23:18:26Z  
> verb="Identify">http://export.arxiv.org/oai2arXivhttp://export.arxiv.org/oai22.0h...@arxiv.org2007-05-23persistent-MM-DD  xmlns="http://www.openarchives.org/OAI/1.1/eprints"; 
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
> xsi:schemaLocation="http://www.openarchives.org/OAI/1.1/eprints 
> http://www.openarchives.org/OAI/1.1/eprints.xsd";>   Author 
> self-archived e-printsMetadata 
> harvesting permitted through OAI interface  
> http://arxiv.org/help/oa/metadataPolicy  
>   Full-content harvesting not permitted (except by special 
> arrangement)  http://arxiv.org/help/oa/dataPolicy 
>Author self-submission preferred, 
> submissions screened for appropriateness.  
> http://arxiv.org/help/submit 
>  xmlns="http://www.openarchives.org/OAI/2.0/branding/"; 
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
> xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/branding/ 
> http://www.openarchives.org/OAI/2.0/branding.xsd";>  
> http://arxiv.org/OAI/arXivLogo.png http://arxiv.org/ 
> arXiv e-print archive 88 31 
>  metadataNamespace="http://www.openarchives.org/OAI/2.0/oai_dc/"; 
> mimeType="text/xsl">http://arxiv.org/OAI/arXivMetadata.xsl
>
> ```
>
[2] http://docs.basex.org/wiki/Web_Module

Re: [basex-talk] Understanding the path index

[Bridger Dyson-Smith]

Julia, all,

apologies - I hit 'send' a bit too quickly. I wanted to ask how you were
ingesting RDF, what the size of your input looked like, and how you were
dealing with namespaces. I have some RDF/MADS data from the Library of
Congress and I've never been able to ingest it into BaseX, even stripping
namespaces, etc, things seem to run out of memory or stall.

Would you be willing to share some details about your data?

Thanks very much.
Best,
Bridger

On Fri, Oct 25, 2019 at 3:41 PM Bridger Dyson-Smith 
wrote:

> Hi Julia -
>
> Preface: let me be clear when I say that I've wondered about some of this
> myself, so I don't think I have an answer for you. That being said, I
> wonder if this is a grouping/data modeling problem: i.e. you have 5,000
> aggregations refer to 1 of 3 web resources vs ~7,000 aggregations each with
> their own distinct web resource.
>
> If you created 3 databases for "abc" ( hm... "a", "b", and "c"? ☺), one
> for each web resource (i.e. where there would be only
> aggregations-to-the-specific-web-resource), would that help with query
> times at all? It might necessitate a bit of pre-processing in your creation
> step though.
>
> In any event, I hope those random thoughts are helpful in some way.
> Best,
> Bridger
>
>
> On Fri, Oct 25, 2019 at 10:24 AM Beck, Julia 
> wrote:
>
>> Hi,
>>
>> first of all: thank you, the fix for [1] did the trick and in 9.2.4 the
>> query is working as expected.
>> Today, I come back to you with another challenge in performance which
>> again seems to have something to do with indexing(?). So here's the
>> situation:
>> I have two databases "abc" and "def". "abc" contains 1 xml doc with
>> about 150.000 nodes and "def" contains 1 xml doc with about 400.000
>> nodes. Both are similarly strutured and have their up-to-date text and
>> attr indexes. The xml docs look both (simplified) like the following:
>>
>> 
>>   
>>  
>>  <...>
>>   
>>   
>>  <...>
>>   
>>   
>>  
>>  <...>
>>   
>>   
>>  <...>
>>   
>>   
>>  
>>  
>>  <...>
>>   
>>   
>>  <...>
>>   
>>   <...>
>> 
>>
>> So one aggregation refers to one (or more) web resources. I boiled down
>> my original query to the following purpose to keep it simple: for each
>> aggregation give me the corresponding web resource.
>>
>>for $agg in db:open($db_name)/rdf:RDF/ore:Aggregation
>>return for $urn in $agg/edm:object/@rdf:resource
>>  return (# db:enforceindex #)
>>{db:open($db_name)/rdf:RDF/edm:WebResource[@rdf:about=$urn]}
>>
>> For both databases the query gives me the required result and the query
>> info tells me that the attribute index for $urn is applied in both
>> cases (this is also the case if I leave out the pragma). However, oddly
>> enough, for the "larger" database "def" with a larger attribute index
>> it takes roughly 1 second while the "smaller" database "abc" with a
>> smaller attribute index takes 20 seconds. This is not very long but the
>> original query is more complicated and I have bigger databases with the
>> same structure where it starts to matter.
>>
>> The only (and I think important) difference between "abc" and "def" is
>> that "abc" contains only 3 web resources that all 5.000 aggregations
>> refer to. While in "def" each aggregation refers to a particular web
>> resource (== 7.000 aggregations and 7.000 web resources).
>>
>> With index:facets I had a look at the facet values and learned that
>> there is a "maximum number of distinct values to store per name". Is
>> there a difference in performance because of that? Maybe I do not get
>> the index structures but it feels strange that it takes longer to find
>> the correct attribute in a range of 3 different values than in a range
>> of 7.000. Maybe there is also another problem in my query, databases or
>> my reasoning that I do not see? Either way, I need help in
>> understanding this phenomenon :-)
>>
>> I hope you could follow, please don't hesitate to ask if you need
>> anything to reproduce this situation (I am using BaseX 9.2.4).
>>
>> Julia
>>
>> [1]
>> https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-July/014511.html
>>
>

Re: [basex-talk] Understanding the path index

[Bridger Dyson-Smith]

Hi Julia -

Preface: let me be clear when I say that I've wondered about some of this
myself, so I don't think I have an answer for you. That being said, I
wonder if this is a grouping/data modeling problem: i.e. you have 5,000
aggregations refer to 1 of 3 web resources vs ~7,000 aggregations each with
their own distinct web resource.

If you created 3 databases for "abc" ( hm... "a", "b", and "c"? ☺), one for
each web resource (i.e. where there would be only
aggregations-to-the-specific-web-resource), would that help with query
times at all? It might necessitate a bit of pre-processing in your creation
step though.

In any event, I hope those random thoughts are helpful in some way.
Best,
Bridger


On Fri, Oct 25, 2019 at 10:24 AM Beck, Julia 
wrote:

> Hi,
>
> first of all: thank you, the fix for [1] did the trick and in 9.2.4 the
> query is working as expected.
> Today, I come back to you with another challenge in performance which
> again seems to have something to do with indexing(?). So here's the
> situation:
> I have two databases "abc" and "def". "abc" contains 1 xml doc with
> about 150.000 nodes and "def" contains 1 xml doc with about 400.000
> nodes. Both are similarly strutured and have their up-to-date text and
> attr indexes. The xml docs look both (simplified) like the following:
>
> 
>   
>  
>  <...>
>   
>   
>  <...>
>   
>   
>  
>  <...>
>   
>   
>  <...>
>   
>   
>  
>  
>  <...>
>   
>   
>  <...>
>   
>   <...>
> 
>
> So one aggregation refers to one (or more) web resources. I boiled down
> my original query to the following purpose to keep it simple: for each
> aggregation give me the corresponding web resource.
>
>for $agg in db:open($db_name)/rdf:RDF/ore:Aggregation
>return for $urn in $agg/edm:object/@rdf:resource
>  return (# db:enforceindex #)
>{db:open($db_name)/rdf:RDF/edm:WebResource[@rdf:about=$urn]}
>
> For both databases the query gives me the required result and the query
> info tells me that the attribute index for $urn is applied in both
> cases (this is also the case if I leave out the pragma). However, oddly
> enough, for the "larger" database "def" with a larger attribute index
> it takes roughly 1 second while the "smaller" database "abc" with a
> smaller attribute index takes 20 seconds. This is not very long but the
> original query is more complicated and I have bigger databases with the
> same structure where it starts to matter.
>
> The only (and I think important) difference between "abc" and "def" is
> that "abc" contains only 3 web resources that all 5.000 aggregations
> refer to. While in "def" each aggregation refers to a particular web
> resource (== 7.000 aggregations and 7.000 web resources).
>
> With index:facets I had a look at the facet values and learned that
> there is a "maximum number of distinct values to store per name". Is
> there a difference in performance because of that? Maybe I do not get
> the index structures but it feels strange that it takes longer to find
> the correct attribute in a range of 3 different values than in a range
> of 7.000. Maybe there is also another problem in my query, databases or
> my reasoning that I do not see? Either way, I need help in
> understanding this phenomenon :-)
>
> I hope you could follow, please don't hesitate to ask if you need
> anything to reproduce this situation (I am using BaseX 9.2.4).
>
> Julia
>
> [1]
> https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-July/014511.html
>

Re: [basex-talk] http:request parsing question

[Bridger Dyson-Smith]

Hi Liam!

On Tue, Oct 22, 2019 at 7:38 PM Liam R. E. Quin 
wrote:

> On Tue, 2019-10-22 at 19:32 -0400, Bridger Dyson-Smith wrote:
> >
> > http://export.arxiv.org/oai2?verb=Identify'/>)//@status/data()
> >
> > returns '200', but trying
> >
> > http:send-request()/h:response/@status
> >
> > fails.
>
> I'm guessing that the first request sends you an auth token, and that
> you're supposed to send this as a parameter in the second token?
>
> How exactly does the second call fail?
>
> Apologies for not clarifying. The @href points to an OAI-PMH endpoint,
which simply returns a XML response[1] saying, "This is an OAI-PMH
endpoint, here's an administrator's email address, plus some other stuff."
So, there aren't any tokens being passed back in forth in this case (to be
honest, there are resumption tokens that can be passed back in to an OAI
provider for other types of requests).

I had thought it might be a good idea to try to think about the HTTP status
('200', '404', etc) in a thing I'm working on, and while I realize I can
get to it with a `//@status` , I had thought a more direct expression would
be more suited (what if there are other `@status` attributes, etc). I even
tried taking my shoes off, to no avail. :)

In any case, thank you!
Best,
Bridger


> --
> Liam Quin, https://www.delightfulcomputing.com/
> Available for XML/Document/Information Architecture/XSLT/
> XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
> Barefoot Web-slave, antique illustrations:  http://www.fromoldbooks.org
>
> [1] The full response from arXiv's OAI endpoint looks like this:
```

http://www.openarchives.org/OAI/2.0/";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/
http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd";>2019-10-22T23:18:26Zhttp://export.arxiv.org/oai2arXivhttp://export.arxiv.org/oai22.0h...@arxiv.org2007-05-23persistent-MM-DDhttp://www.openarchives.org/OAI/1.1/eprints";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.openarchives.org/OAI/1.1/eprints
http://www.openarchives.org/OAI/1.1/eprints.xsd";> 
Author self-archived e-prints  
 Metadata harvesting permitted through OAI interface
http://arxiv.org/help/oa/metadataPolicy 
  Full-content harvesting not permitted (except by
special arrangement)
http://arxiv.org/help/oa/dataPolicy 
  Author self-submission preferred,
submissions screened for appropriateness.
http://arxiv.org/help/submit
http://www.openarchives.org/OAI/2.0/branding/";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/branding/
http://www.openarchives.org/OAI/2.0/branding.xsd";> 
http://arxiv.org/OAI/arXivLogo.png
http://arxiv.org/ arXiv e-print archive
88 31
http://www.openarchives.org/OAI/2.0/oai_dc/";
mimeType="text/xsl">http://arxiv.org/OAI/arXivMetadata.xsl

```

[basex-talk] http:request parsing question

[Bridger Dyson-Smith]

Hi all -

I was hoping that someone on the list could help with my thinking about
parsing the http:response values from a request; I've tried the following,
but I'm not thinking about this correctly.

declare namespace oai = "http://www.openarchives.org/OAI/2.0/";;
declare namespace h = "http://expath.org/ns/http-client";;
http:send-request()//@status/data()

returns '200', but trying

http:send-request()/h:response/@status

fails.

I'm guessing (but am not sure) that the combination of the 
with the body of the response is causing my problem, but I'm not sure how
to wiggle my way around it.

Note: this is not in a RESTXQ context (yet), but for my simple trials is
there a way to get at the http -namespace elements?
Thank you for your time and trouble.
Best,
Bridger

Re: [basex-talk] simplest possible FLOWR specifying a database

[Bridger Dyson-Smith]

Hi thufir,

The specification might be helpful here [1,2], but essentially it is the
thing being processed, given as a sort of global scope.

HTH!
Best,
Bridger

[1] https://www.w3.org/TR/xquery-31/#id-context-item-expression
[2] https://www.w3.org/TR/xquery-31/#dt-context-item

On Sat, Oct 12, 2019, 4:49 PM thufir  wrote:

> that's interesting. what's item in this example, and how can it be
> referenced?
>
>
> thanks,
>
> Thufir
>
> On 2019-10-12 2:29 a.m., Christian Grün wrote:
> > On top of the query (in the query prolog), you can bind your database
> > to the context. After that, there’ll be no need to bind it to a
> > variable:
> >
> > declare context item := db:open("com.w3schools.books");
> > /bookstore
> >
> >
> >
> > On Sat, Oct 12, 2019 at 10:11 AM thufir  wrote:
> >>
> >> nevermind, it's:
> >>
> >> let $db := db:open("com.w3schools.books")
> >> return $db/bookstore
> >>
> >> I thought it was unusual because of db:open.
> >>
> >>
> >> -Thufir
> >>
> >> On 2019-10-12 12:50 a.m., thufir wrote:
> >>> these FLWOR .xq files work as is:
> >>>
> >>> let $db := db:open("com.w3schools.books")
> >>> for $x in $db/bookstore
> >>> return $x
> >>>
> >>>
> >>> or
> >>>
> >>> {
> >>> let $db := db:open("com.w3schools.books")
> >>> for $x in $db/bookstore/book
> >>> return {$x/title,$x/author}
> >>> }
> >>>
> >>>
> >>> Probably a silly question, but, rather than, as in the first query,
> >>> using $x can I not somehow specify just "/"?
> >>>
> >>>
> >>>
> >>> thanks,
> >>>
> >>> Thufir
>

Re: [basex-talk] binding-types?

[Bridger Dyson-Smith]

Hi Ben -

Does something like:
```
declare variable $name external; for $i as xs:integer in 1 to 5 return
element { $name } { $i }
```
work any differently?

Best,
Bridger

On Thu, Sep 12, 2019 at 10:49 AM ben.engb...@be-logical.nl <
ben.engb...@be-logical.nl> wrote:

> xs:string is neither accepted and says that no namespace is defined for
> 002xs:string
>
> Ben
>
>
>  Oorspronkelijk bericht 
> Onderwerp: [basex-talk] binding-types?
> Van: Ben Engbers
> Aan: Basex Mail-lijst
> Cc:
>
>
> Hi,
>
> While creating a R-package, based on my R client-implementation, I found
> that the binding function was malfunctioning.
>
> After rewriting that function, the following is accepted:
> query_txt <- "declare variable $name external; for $i in 1 to 5 return
> element { $name } { $i }"
> query_obj_1 <- Query(Sess, query_txt)
> success <- query_obj_1$queryObject$Bind("name", "number")
> print(query_obj_1$queryObject$ExecuteQuery())
>
> results in:
> "1" "2" "3"
> "4" "5"
>
> When I change the line
> 'success <- query_obj_1$queryObject$Bind("name", "number")' in
> success <- query_obj_1$queryObject$Bind("name", "number", "xs:integer")
> the following error is produced:
> "[XPST0081] No namespace declared for '\002xs:integer'."
>
> What types are accepted?
>
> Ben
>
>

Re: [basex-talk] Passing through entities unchanged when serializing

[Bridger Dyson-Smith]

Ha ha, awesome Liam! Thank you for clarifying!

Best,
Bridger


On Mon, Sep 9, 2019 at 9:37 PM Liam R. E. Quin 
wrote:

> On Tue, 2019-09-10 at 02:59 +0200, Andreas Mixich wrote:
> > I wonder why the serialization behaves that way. It does not make
> > sense to
> > me. If a user has the need to escape XML, it should be thorough,
> > shouldn't it?
>
> XML entities are expanded by he XML parser, so by the time XQuery (or
> XSLT) sees the document they are gone.
>
> Consider an entity like
>  "blackgreySteven on>">
>
> &boy
>
> It'd be really complex to have that visible to XPath and to have to
> write, e.g.
> /students/entity(*)/person
>
> If it's an external parsed entity it's visible in that the base-uri
> property changes, but that's all.
>
> Character entities like &rcedilla; (ŗ) are just special cases of
> general entities, and XML does not distinguish them. I wish it did, but
> we never got back to that work after publishing XML 1.0.
>
> Liam
>
> --
> Liam Quin, https://www.delightfulcomputing.com/
> Available for XML/Document/Information Architecture/XSLT/
> XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
> Web slave for vintage clipart http://www.fromoldbooks.org/
>
>

Re: [basex-talk] Passing through entities unchanged when serializing

[Bridger Dyson-Smith]

Hi Andreas -
I'm not sure (way outside of my wheelhouse :), but I think because
arbitrary serialization can generate invalid XML, so having a character map
makes the possible invalidity explicit?
Now that I've typed that, I'm not sure if that captures the rational or
not. :) In any case, here's what the specifications have to say[1].

Best,
Bridger

[1] https://www.w3.org/TR/xslt-xquery-serialization-31/#character-maps



On Mon, Sep 9, 2019 at 9:00 PM Andreas Mixich 
wrote:

> I wonder why the serialization behaves that way. It does not make sense to
> me. If a user has the need to escape XML, it should be thorough, shouldn't
> it?
>
> On Mon, Sep 9, 2019 at 10:47 PM Liam R. E. Quin 
> wrote:
>
>> On Mon, 2019-09-09 at 15:04 +0200, Andreas Mixich wrote:
>> > when serializing a string, that contains literal XML with entities,
>> > how do I pass through those entities unchanged?
>>
>> One way is to use a character map, as Bridger Dyson-Smith described.
>>
>> Sometimes another way can be to have a version of the DTD in which the
>> replacement text of the entity marks the presence of the entity, e.g.
>> 
>> but this will affect full-text searching of course.
>>
>> Liam
>>
>> --
>> Liam Quin, https://www.delightfulcomputing.com/
>> Available for XML/Document/Information Architecture/XSLT/
>> XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
>> Barefoot Webslave for old illustrations  http://www.fromoldbooks.org/
>>
>>
>
> --
> Minden jót, all the best, Alles Gute,
> Andreas Mixich
>

Re: [basex-talk] Passing through entities unchanged when serializing

[Bridger Dyson-Smith]

Hi Andreas -

Have you tried using different serialization options? I.e.,
serialize.xq:
```
declare option output:method "xml";
declare option output:parameter-document "map.xml";
declare variable $input := "Lorem ipsum, ' dolor sit amet.";
serialize($input)
```

map.xml:
```
http://www.w3.org/2010/xslt-xquery-serialization";>
  

  

```

When run in the BaseX GUI, I get:
`<p>Lorem ipsum, ' dolor sit amet.</p>`,
might be closer?

I think you might have been experiencing the default 'basex' serialization
option (see [1] for more).
Hope that helps.
Best,
Bridger

[1] http://docs.basex.org/wiki/Serialization

On Mon, Sep 9, 2019 at 9:05 AM Andreas Mixich 
wrote:

> Hi,
>
> when serializing a string, that contains literal XML with entities, how do
> I pass through those entities unchanged?
> Example:
>
> let $input := "Lorem ipsum ' dolor sit amet "
> return serialize($input)
>
> results in:
>
> 
Lorem ipsum dolor sit amet, ' consectetur adipisicing
> elit.
>
> but I want:
>
> 
Lorem ipsum dolor sit amet, ' consectetur adipisicing
> elit.
>
> --
> Minden jót, all the best, Alles Gute,
> Andreas Mixich
>

Re: [basex-talk] Processing instruction constructors: problem with output

[Bridger Dyson-Smith]

Hi Tim -
A quick test and this is working correctly(?) for me;
1. I cloned Alain's repo to `.../webapp/static/`
2. I copy-pasta'd your example into `.../webapp/test.xqm`
3. After starting the BaseX HTTP server, I hit `localhost:8984` and
`localhost:8984/test` and had good responses from both; i.e. at `/test`,
I'm seeing a "Test", a "Placeholder", and a fun little gray-ish ball (but
no "3" in either browser).

This was with Firefox 60.8.0esr and Chromium 76.0.3809.87.

Is there anything specific I can try to tell you? I'm not very familiar
with XSLTforms.
Best,
Bridger

On Thu, Aug 22, 2019 at 11:16 AM Christian Grün 
wrote:

> I can’t promise if I can further dive into this. Maybe Alain has some
> hints?
>
>
> On Thu, Aug 22, 2019 at 5:09 PM Tim Thompson  wrote:
> >
> > Okay, so the response panel shows the correct output, but for some
> reason the stylesheet is not being called. If it's not too much trouble,
> would you be able to try this with XSLTForms (
> https://github.com/AlainCouthures/xphoneforms.git) placed in the BaseX
> webapp static dir?
> >
> > Here is an expanded demo query:
> >
> >
> > xquery version "3.1";
> >
> > module namespace test = "test";
> >
> > declare
> >   %rest:path("/test")
> >   %rest:GET
> >   %output:method("xhtml")
> > function test:home(
> >
> > ) as document-node() {
> >
> >   document {
> > processing-instruction { "xml-stylesheet" } {
> > 'type="text/xsl"
> href="/static/xphoneforms/build/xsl/xsltforms.xsl"'
> > },
> > processing-instruction { "xsltforms-options" } { 'debug="no"' },
> > processing-instruction { "css-conversion" } { 'no' }
> > ,
> > http://www.w3.org/1999/xhtml"; xmlns:xf="
> http://www.w3.org/2002/xforms";
> >   xmlns:ev="http://www.w3.org/2001/xml-events"; xmlns:my="test"
> >   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xmlns:xsd="http://www.w3.org/2001/XMLSchema";>
> >
> >   
> > TALLER BIBFRAME
> > http://www.w3.org/2002/xforms"; id="main">
> >   
> > 
> >   Test
> > 
> >   
> > 
> >
> >
> >   
> >   
> > Placeholder
> > 
> > 
> >   
> > 
> >   
> > 
> >   }
> >
> > };
> >
> > --
> > Tim A. Thompson
> > Discovery Metadata Librarian
> > Yale University Library
> >
> > On Thu, Aug 22, 2019 at 10:57 AM Christian Grün <
> christian.gr...@gmail.com> wrote:
> >>
> >> Hm… You could check the Response panel in the Network View of your
> >> browser. The latest version of Chrome, Firefox, Opera, Edge and IE
> >> should give you the correct result (those are the ones that I have
> >> tried).
> >>
> >>
> >> > I'm getting the comments in the browser (web console). When I
> dereference the endpoint with curl, the processing instructions are correct.
> >> >
> >> > In the browser, the XForms instructions are not being processed. I
> don't think XSLTForms would be the culprit, because the stylesheet is not
> being called.
> >> >
> >> > --
> >> > Tim A. Thompson
> >> > Discovery Metadata Librarian
> >> > Yale University Library
> >> >
> >> > On Thu, Aug 22, 2019 at 10:37 AM Christian Grün <
> christian.gr...@gmail.com> wrote:
> >> >>
> >> >> Hi Tim,
> >> >>
> >> >> I have just tested your endpoint code (with the latest version of
> >> >> BaseX), and I receive processing instructions as expected:
> >> >>
> >> >>  >> >> href="static/xphoneforms/build/xsl/xsltforms.xsl"?>
> >> >> ...
> >> >>
> >> >> Which version of BaseX are you using? Are you also getting comments
> in
> >> >> your response when calling the function from the browser or via other
> >> >> tools (such as curl), and could it have to do with the way how
> >> >> XSLTForms processes the result?
> >> >>
> >> >> Best,
> >> >> Christian
> >> >>
> >> >>
> >> >> On Thu, Aug 22, 2019 at 4:31 PM Tim Thompson 
> wrote:
> >> >> >
> >> >> > Hello,
> >> >> >
> >> >> > I'm trying to output an XForms (XSLTForms) document from a RESTXQ
> endpoint, but the necessary processing instructions are being serialized in
> XML comments in the output, e.g.:
> >> >> >
> >> >> > 
> >> >> >
> >> >> >  Am I missing something?
> >> >> >
> >> >> > XQuery:
> >> >> >
> >> >> > xquery version "3.1";
> >> >> >
> >> >> > module namespace test = "test";
> >> >> >
> >> >> > declare
> >> >> >   %rest:path("/test")
> >> >> >   %rest:GET
> >> >> >   %output:method("xhtml")
> >> >> > function test:home() as document-node() {
> >> >> >
> >> >> >   document {
> >> >> > processing-instruction { "xml-stylesheet" } {
> >> >> > 'type="text/xsl"
> href="static/xphoneforms/build/xsl/xsltforms.xsl"'
> >> >> > },
> >> >> > processing-instruction { "xsltforms-options" } { 'debug="no"'
> },
> >> >> > processing-instruction { "css-conversion" } { 'no' }
> >> >> > ,
> >> >> > http://www.w3.org/1999/xhtml"; xmlns:xforms="
> http://www.w3.org/2002/xforms";
> >> >> >   xmlns:ev="http://www.w3.org/2001/xml-events";
> xmlns:my="test"

Re: [basex-talk] XQuery: Subsequence of a sequence

[Bridger Dyson-Smith]

Hi Jordan,

Maybe the fn:head(), fn:tail(), and other functions for sequence operations
would help[1]?

HTH!
Best,
Bridger

[1] https://www.w3.org/TR/xpath-functions-31/#sequence-functions

On Tue, Jul 2, 2019, 6:27 PM Jordan Castillo 
wrote:

> BaseX team,
>
> Sorry for the long question but I just figured out what is going on...
>
> The '$stereotypes' is a gigantic sequence, which is why I can't separate
> its members into sequences.
>
> Hope you guys have a great week,
>
> Jordan Castillo
>
> On Tue, Jul 2, 2019 at 3:05 PM Jordan Castillo <
> jordantcastillo1...@gmail.com> wrote:
>
>> Hello BaseX Staff,
>>
>> I am trying to access members of a sequence that I have created already,
>> but it whenever I use the functions subsequence() it returns the entire
>> sequence, and count() always returns 1.
>>
>>
>>
>> The code:
>> xquery version "3.1";
>> declare namespace map="http://www.w3.org/2005/xpath-functions/map";;
>> declare namespace xmi="http://www.omg.org/spec/XMI/20131001";;
>> declare namespace uml="http://www.omg.org/spec/UML/20131001";;
>> declare namespace MagicDraw_Profile="
>> http://www.omg.org/spec/UML/20131001/MagicDrawProfile";;
>> declare namespace Extra_stereotypes="
>> http://www.magicdraw.com/schemas/Extra_stereotypes.xmi";;
>> declare namespace UAF="http://www.omg.org/spec/UPDM/20121004/UPDM-Profile
>> ";
>> (
>>   let $begin :=
>> db:open("CubeSatUML2.5XMIv3")/xmi:XMI/uml:Model/packagedElement
>>   let $modelContent := $begin//packagedElement[@xmi:type="uml:Class"]
>>   (: Combine the names of the elements with their ID's :)
>>   let $names := $modelContent/@name
>>   let $ids := $modelContent/@xmi:id
>>   (: Grab all stereotypes, properties:[attributes, comments] :)
>>   let $stereotypes := for $id in $ids
>> let $stereo1 := /xmi:XMI/*[@base_Element=$id]
>> let $stereo2 := /xmi:XMI/*[@base_Class=$id]
>> let $attributes := $begin//*[@xmi:id=$id]/ownedAttribute/[@name |
>> @xmi:type | @xmi:id]
>> let $comments := $begin//*[@xmi:id=$id]/ownedComment/[@name |
>> @xmi:type | @xmi:id | @body]
>> let $combined := ($id, $stereo1/name(), $stereo2/name(), $comments,
>> $attributes, "")
>> return $combined
>>
>>   let $final :=
>> for $seq in $stereotypes
>>   return subsequence($seq, 1, 1)
>> (:  ^ the part in question:)
>>
>>   return $final
>>   )
>>
>> I realize I am creating this in a very weird way, but I want to keep the
>> data together and have not thought of another solution.
>> I have tried 'for each'ing over the '$sequences' and tokenizing to get
>> the elements out, but I get an error:
>>   let $finalMap :=
>> for $seq in $stereotypes
>>   return tokenize($seq, '\s')
>> [XPTY0004] Item expected, sequence found: ("uml:Property",
>> "_18_5_2_4c1014b_153850
>>
>>
>> When I print '$stereotypes', I get what appears to be a sequence of
>> sequences (which I don't think is a real thing, but it appears to be that)
>>
>> Ex output:
>>
>> xmi:id="_18_5_2_4c1014b_1538500371989_609286_80522"
>> UAF:ResourceArtifact
>> [(xmi:type="uml:Property",
>> xmi:id="_18_5_2_4c1014b_1538500468390_289646_80939", name="flight")]
>> [(xmi:type="uml:Property",
>> xmi:id="_18_5_2_4c1014b_1538500488269_847951_81240", name="ground")]
>>
>> xmi:id="_18_5_2_4c1014b_1538500376501_394659_80590"
>> UAF:ResourceArtifact
>> [(xmi:type="uml:Comment",
>> xmi:id="_18_5_2_4c1014b_1538501028725_896600_82978", body="The ground
>> supporting equipment and operations that support the spacecraft mission.")]
>> [(xmi:type="uml:Property",
>> xmi:id="_18_5_2_4c1014b_1538500492755_533021_81326", name="ground
>> Communications")]
>> [(xmi:type="uml:Property",
>> xmi:id="_18_5_2_4c1014b_1538500495333_439654_81369", name="ground Command
>> and Control")]
>> [(xmi:type="uml:Port",
>> xmi:id="_18_5_2_4c1014b_1538582001724_435913_83660", name="Flight_Ground")]
>>
>> My end goal is to have a map of maps, where the root map has keys of
>> element names (currently in $names) and the value is a map of all the
>> properties of this element.
>> To do this I have to deconstruct this monster I have created, and maybe I
>> am on the wrong path entirely but some direction would be greatly
>> appreciated.
>>
>> End goal format:
>>
>> map{
>> elementName : map {
>> stereotypes: map{
>> stereotype_Name_1: stereotype_ID_1,
>> stereotype_Name_2: stereotype_ID_2
>> },
>> attributeName: map{
>> 'id': actual_ID_here,
>> 'lowerValue': someValue,
>> 'upperValue': "*"
>> },
>> comment: map{
>> 'id': actual_ID_here,
>> 'xmi:type': 'uml:Comment',
>> 'body': actual_Comment_Here
>> },
>> 'id': actual_ID_here
>> }
>> }
>>
>> Thanks,
>>
>> Jordan Castillo
>> 
>>
>>

[basex-talk] Query Optimization Question

[Bridger Dyson-Smith]

Hi all -

An interesting XQuery question came up on reddit[1] over the weekend, and
I'm curious about a couple of aspects of the problem.

To recreate, the sample data is at timecenters[2] in the
employeeTemporalDataset.zip download (specifically, the `departments.xml`
and the three compressed directories of XML in `employees.tar.gz`). I
created a database from employees.tar.gz (thanks for letting us parse XML
in archives!), added the `departments.xml`, and then `Optimized All`.

The initial query was
```
for $emp in /employees/employee[@tend = '-01-01']
let $curdept := $emp/deptno[@tend = '-01-01']
return
  $emp/lastname || " " || $curdept || " " || /departments/department[deptno
= $curdept]/deptname
```
and it is slow (~100 minutes, 5999201.28 ms). The original poster came back
later with a modified query that is significantly faster[3], but I was
mostly wondering about the whys of the slowness. I think (but am not sure)
that this is a join, and since the initial `for` binding ($emp) is pretty
big (~240K), the processor has to parse through the $emp result sequence
and match to values in the /departments part of the database; is that a
correct assumption?

Again, I'm assuming that in the second, faster, query, the processor has
two sequences ($d and $e) and is able to pull the joined data together much
more quickly?

Thanks in advance for any light you can shed on these questions.
Best,
Bridger

[1] https://www.reddit.com/r/xml/comments/c3mb86/simple_xquery_optimization/
[2] http://timecenter.cs.aau.dk/software.htm
[3] improved query:
```
for $d in /departments/department
for $e in /employees/employee[deptno[@tend='-01-01'] = $d/deptno]
return
  $e/lastname ||  " " || $d/deptno || " " || $d/deptname
```

Re: [basex-talk] Extracting function signatures from documentation

[Bridger Dyson-Smith]

Hi Andreas -

(someone on the BaseX team will surely correct where I get this wrong,
so...) as far as I know, many/most/all of the BaseX builtin modules are
written in Java, not XQuery, so I don't know that there are modules to
inspect, so to speak. However, late last year there was some discussion on
the list along these same lines and Christian wrote up an XQuery script to
generate xqDoc output from the wiki [1].

Maybe that would be helpful for your project?
Best,
Bridger

[1] https://github.com/BaseXdb/basex-dist/blob/master/wiki2xqdoc.xq



On Thu, May 16, 2019 at 2:34 PM Andreas Mixich 
wrote:

> On Thu, May 16, 2019 at 5:54 PM Loren Cahlander 
> wrote:
>
>> Please take a look at xqDoc for XQuery that is like JavaDoc for Java.
>>
>> https://github.com/lcahlander/xqdoc
>>
>>
> Thanks for the hint! I did not yet think about looking at xqdoc.
> Definitely will do this. See also my reply to Christian.
>
> --
> Minden jót, all the best, Alles Gute,
> Andreas Mixich
>