Re: [basex-talk] BaseX 10.7: The Summer Edition

[Tim Thompson]

Congrats on the latest version! Looking forward as usual to exploring the
new features.

However, I'm perplexed by the decision to remove the text parser from the
codebase. I understand the desire to streamline and remove dependencies
related to lower-value features, but I've always found the text parser to
be super useful. After installing Basex 10.8 beta today, I had to refactor
a process (parsing a set of interview transcripts generated by Zoom) that
involved creating a DB from a directory of text files.

In addition, I noticed some unexpected results in how the text was parsed
using standard methods. In BaseX 10.6, using the text parser in the GUI,
the output looks like this:

WEBVTT

1
00:00:02.910 -- 00:00:27.240
...


Here, each line end is just a newline character (\n).

Using file:read-text or fn:unparsed-text (in 10.6 and 10.8 beta), the
output looks like this:

WEBVTT

1
00:00:02.910 -- 00:00:27.240
...


Here, each line end also has a carriage return (\r).

And if instead, I store it as an XQuery value, I see the newline characters
that aren't otherwise displayed in the GUI:

"WEBVTT100:00:02.910 -->
00:00:27.240..."

So, the text parser seems to have done some normalization, which was also
helpful.

Any chance that it could be restored (by popular demand) in version 11? :)

Best regards,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library

El vie, 4 ago 2023 a la(s) 06:55, Christian Grün (christian.gr...@gmail.com)
escribió:

> Dear all,
>
> We’re pleased to announce version 10.7 of BaseX, of our XML framework:
> https://basex.org.
>
> The new release is a big step forward towards BaseX 11:
>
> • We have added numerous new operators, functions and features of XQuery
> 4.
> • The GUI editor and result view now provide full support for Unicode
> characters.
> • Font rendering has been improved (you can tweak it in the Font Dialog)
> • More BaseX 11 preview features available (see docs.basex.org)
> • Various bug fixes (web:forward, job:eval, main-memory documents)
>
> Have fun,
> Your BaseX Team
>

Re: [basex-talk] Working with custom stores

[Tim Thompson]

Sure, attached are two command scripts: one that fails (when dumping store
contents) and one that passes (when adding a value directly).

Thanks,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
www.linkedin.com/in/timathompson
timothy.thomp...@yale.edu


El mar, 15 nov 2022 a la(s) 13:21, Christian Grün (christian.gr...@gmail.com)
escribió:

> Wow. I’ll look into that soon. Do you think you can provide me with a
> little test case (e.g., a BaseX command script) to reproduce this?
>
>
>
>
> Tim Thompson  schrieb am Di., 15. Nov. 2022, 19:16:
>
>> Thanks! I just tried to dump the contents of the store to a database
>> value:
>>
>> db:put-value(
>>   "entries",
>>   array {
>> store:keys() ! store:get(.) (: sequence of arrays :)
>>   },
>>   "x"
>> )
>>
>> But when I query with db:get-value("entries", "x"), I get a "Not
>> Expected" error. Should this kind of operation be possible?
>>
>> Error:
>> Improper use? Potential bug? Your feedback is welcome:
>> Contact: basex-talk@mailman.uni-konstanz.de
>> Version: BaseX 10.3 beta
>> Java: Azul Systems, Inc., 11.0.16
>> OS: Mac OS X, aarch64
>> Stack Trace:
>> java.lang.RuntimeException: Not Expected.
>> at org.basex.util.Util.notExpected(Util.java:54)
>> at org.basex.query.value.type.AtomType.read(AtomType.java:983)
>> at org.basex.core.Store.read(Store.java:315)
>> at org.basex.query.func.db.DbGetValue.resource(DbGetValue.java:70)
>> at org.basex.query.func.db.DbGetValue.value(DbGetValue.java:46)
>> at org.basex.query.func.db.DbGetValue.value(DbGetValue.java:27)
>> at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46)
>> at org.basex.query.scope.MainModule.iter(MainModule.java:48)
>> at org.basex.query.QueryContext.lambda$4(QueryContext.java:354)
>> at org.basex.query.QueryContext.run(QueryContext.java:763)
>> at org.basex.query.QueryContext.iter(QueryContext.java:354)
>> at org.basex.query.QueryContext.lambda$6(QueryContext.java:601)
>> at org.basex.query.QueryContext.run(QueryContext.java:763)
>> at org.basex.query.QueryContext.cache(QueryContext.java:598)
>> at org.basex.query.QueryProcessor.cache(QueryProcessor.java:118)
>> at org.basex.core.cmd.AQuery.run(AQuery.java:82)
>> at org.basex.core.Command.run(Command.java:233)
>> at org.basex.core.Command.execute(Command.java:93)
>> at org.basex.gui.GUI.execute(GUI.java:415)
>> at org.basex.gui.GUI.lambda$5(GUI.java:360)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>>
>>
>>
>> --
>> Tim A. Thompson (he, him)
>> Librarian for Applied Metadata Research
>> www.linkedin.com/in/timathompson
>> timothy.thomp...@yale.edu
>>
>>
>> El mar, 15 nov 2022 a la(s) 13:07, Christian Grün (
>> christian.gr...@gmail.com) escribió:
>>
>>> You'll be safe: Updates with db:put-value will be managed via the
>>> Pending Update List [1], and future queries that operate on the same
>>> database will be queued until the update is finished.
>>>
>>> [1] https://docs.basex.org/wiki/XQuery_Update#Pending_Update_List
>>>
>>>
>>>
>>> Tim Thompson  schrieb am Di., 15. Nov. 2022, 18:50:
>>>
>>>> That's perfect--I knew I was missing something :)
>>>>
>>>> Are db:get-value and db:put-value side-effecting as well?
>>>>
>>>> Thanks,
>>>> Tim
>>>>
>>>>
>>>> --
>>>> Tim A. Thompson (he, him)
>>>> Librarian for Applied Metadata Research
>>>> www.linkedin.com/in/timathompson
>>>> timothy.thomp...@yale.edu
>>>>
>>>>
>>>> El mar, 15 nov 2022 a la(s) 12:28, Christian Grün (
>>>> christian.gr...@gmail.com) escribió:
>>>>
>>>>> Hi Tim,
>>>>>
>>>>> > Using the store module, is it possible to read and merge multiple
>>>>> custom stores into main memory
>>>>>
>>>>> We could add an argument (or options) to file:read to allow entries to
>>>>> be added to an existing main-memory store.
>>>>>
>>>>> > set a particular store as context for a query?
>>>>>
>>>>> The recommended approach is to invoke store:read before any other
>>>>> operation in the query. Please note, however, that the store will be
>>>>> globally available. If a second query runs on the same server and
>>>>> opens another store, it will change the behavior of the first query.
>>>>> For such use cases, it’s better to use databases.
>>>>>
>>>>> > Or, do I need to serialize the stores, e.g., as JSON and then create
>>>>> databases for them?
>>>>>
>>>>> You could have a look at the new db:get-value and db:put-value
>>>>> functions [1]: These functions have particularly been designed to
>>>>> organize XQuery items in databases, making casts to JSON or XML
>>>>> obsolete.
>>>>>
>>>>> Hope this helps,
>>>>> Christian
>>>>>
>>>>> [1] https://docs.basex.org/wiki/Database_Module#db:get-value
>>>>>
>>>>


get-value-test-fails.bxs
Description: Binary data


get-value-test-passes.bxs
Description: Binary data

Re: [basex-talk] Working with custom stores

[Tim Thompson]

Thanks! I just tried to dump the contents of the store to a database value:

db:put-value(
  "entries",
  array {
store:keys() ! store:get(.) (: sequence of arrays :)
  },
  "x"
)

But when I query with db:get-value("entries", "x"), I get a "Not Expected"
error. Should this kind of operation be possible?

Error:
Improper use? Potential bug? Your feedback is welcome:
Contact: basex-talk@mailman.uni-konstanz.de
Version: BaseX 10.3 beta
Java: Azul Systems, Inc., 11.0.16
OS: Mac OS X, aarch64
Stack Trace:
java.lang.RuntimeException: Not Expected.
at org.basex.util.Util.notExpected(Util.java:54)
at org.basex.query.value.type.AtomType.read(AtomType.java:983)
at org.basex.core.Store.read(Store.java:315)
at org.basex.query.func.db.DbGetValue.resource(DbGetValue.java:70)
at org.basex.query.func.db.DbGetValue.value(DbGetValue.java:46)
at org.basex.query.func.db.DbGetValue.value(DbGetValue.java:27)
at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46)
at org.basex.query.scope.MainModule.iter(MainModule.java:48)
at org.basex.query.QueryContext.lambda$4(QueryContext.java:354)
at org.basex.query.QueryContext.run(QueryContext.java:763)
at org.basex.query.QueryContext.iter(QueryContext.java:354)
at org.basex.query.QueryContext.lambda$6(QueryContext.java:601)
at org.basex.query.QueryContext.run(QueryContext.java:763)
at org.basex.query.QueryContext.cache(QueryContext.java:598)
at org.basex.query.QueryProcessor.cache(QueryProcessor.java:118)
at org.basex.core.cmd.AQuery.run(AQuery.java:82)
at org.basex.core.Command.run(Command.java:233)
at org.basex.core.Command.execute(Command.java:93)
at org.basex.gui.GUI.execute(GUI.java:415)
at org.basex.gui.GUI.lambda$5(GUI.java:360)
at java.base/java.lang.Thread.run(Thread.java:829)



-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
www.linkedin.com/in/timathompson
timothy.thomp...@yale.edu


El mar, 15 nov 2022 a la(s) 13:07, Christian Grün (christian.gr...@gmail.com)
escribió:

> You'll be safe: Updates with db:put-value will be managed via the Pending
> Update List [1], and future queries that operate on the same database will
> be queued until the update is finished.
>
> [1] https://docs.basex.org/wiki/XQuery_Update#Pending_Update_List
>
>
>
> Tim Thompson  schrieb am Di., 15. Nov. 2022, 18:50:
>
>> That's perfect--I knew I was missing something :)
>>
>> Are db:get-value and db:put-value side-effecting as well?
>>
>> Thanks,
>> Tim
>>
>>
>> --
>> Tim A. Thompson (he, him)
>> Librarian for Applied Metadata Research
>> www.linkedin.com/in/timathompson
>> timothy.thomp...@yale.edu
>>
>>
>> El mar, 15 nov 2022 a la(s) 12:28, Christian Grün (
>> christian.gr...@gmail.com) escribió:
>>
>>> Hi Tim,
>>>
>>> > Using the store module, is it possible to read and merge multiple
>>> custom stores into main memory
>>>
>>> We could add an argument (or options) to file:read to allow entries to
>>> be added to an existing main-memory store.
>>>
>>> > set a particular store as context for a query?
>>>
>>> The recommended approach is to invoke store:read before any other
>>> operation in the query. Please note, however, that the store will be
>>> globally available. If a second query runs on the same server and
>>> opens another store, it will change the behavior of the first query.
>>> For such use cases, it’s better to use databases.
>>>
>>> > Or, do I need to serialize the stores, e.g., as JSON and then create
>>> databases for them?
>>>
>>> You could have a look at the new db:get-value and db:put-value
>>> functions [1]: These functions have particularly been designed to
>>> organize XQuery items in databases, making casts to JSON or XML
>>> obsolete.
>>>
>>> Hope this helps,
>>> Christian
>>>
>>> [1] https://docs.basex.org/wiki/Database_Module#db:get-value
>>>
>>

Re: [basex-talk] Working with custom stores

[Tim Thompson]

That's perfect--I knew I was missing something :)

Are db:get-value and db:put-value side-effecting as well?

Thanks,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
www.linkedin.com/in/timathompson
timothy.thomp...@yale.edu


El mar, 15 nov 2022 a la(s) 12:28, Christian Grün (christian.gr...@gmail.com)
escribió:

> Hi Tim,
>
> > Using the store module, is it possible to read and merge multiple custom
> stores into main memory
>
> We could add an argument (or options) to file:read to allow entries to
> be added to an existing main-memory store.
>
> > set a particular store as context for a query?
>
> The recommended approach is to invoke store:read before any other
> operation in the query. Please note, however, that the store will be
> globally available. If a second query runs on the same server and
> opens another store, it will change the behavior of the first query.
> For such use cases, it’s better to use databases.
>
> > Or, do I need to serialize the stores, e.g., as JSON and then create
> databases for them?
>
> You could have a look at the new db:get-value and db:put-value
> functions [1]: These functions have particularly been designed to
> organize XQuery items in databases, making casts to JSON or XML
> obsolete.
>
> Hope this helps,
> Christian
>
> [1] https://docs.basex.org/wiki/Database_Module#db:get-value
>

[basex-talk] Working with custom stores

[Tim Thompson]

Hello,

Using the store module, is it possible to read and merge multiple custom
stores into main memory, or to set a particular store as context for a
query? Or, do I need to serialize the stores, e.g., as JSON and then create
databases for them?

Thanks in advance,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
www.linkedin.com/in/timathompson
timothy.thomp...@yale.edu

[basex-talk] Reported memory usage in GUI with Store Module

[Tim Thompson]

Hello,

I've been experimenting with the new Store module, which seems very handy.
I have a custom store that is about 2GB in size. In the GUI (BaseX 10.3
beta), when I try to load it into memory with store:read(), the reported
memory usage appears to max out (with 32GB of available memory):

Total Memory: 32 GB
Reserved Memory: 32 GB
Used Memory: 32 GB

However, monitoring memory usage with htop shows a smaller increase in
memory usage: from 5GB at baseline, it peaks around 15GB, then goes down to
9GB, all while the BaseX GUI appears to be frozen.

After a couple of minutes, the store does load, but GUI performance seems
to be degraded, and it continues to report max memory usage.

Any thoughts?

Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library
www.linkedin.com/in/timathompson

[basex-talk] Indent result for unit tests

[Tim Thompson]

Hello,

I've noticed that the "Indent results" button in the GUI does not seem to
be working when running a test module from the GUI. I'm using BaseX 10.3
beta.

All best,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library
www.linkedin.com/in/timathompson

[basex-talk] Fwd: Available Integration with Apache POI for Excel Generation?

[Tim Thompson]

Resending to the list (forgot to reply all).

-- Forwarded message -
From: Tim Thompson 
Date: Mon, Jul 25, 2022 at 5:04 PM
Subject: Re: [basex-talk] Available Integration with Apache POI for Excel
Generation?
To: Eliot Kimber 


Hi, Eliot,

I'm not aware of any integration with Apache POI (which would be great),
but here's a project--it seems unfinished, but may be enough to get started
with?

https://github.com/eliudmeza/OOXML-Library-XQuery-BaseXdb

All best,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library


On Mon, Jul 25, 2022 at 2:29 PM Eliot Kimber 
wrote:

> I tried searching for “BaseX Excel generation” but it didn’t get me
> farther than a reference to the CSV module, which of course I am familiar
> with.
>
>
>
> I need to add the ability to generate non-trivial Excel spreadsheets,
> which I know how to do using the Apache POI library, but being lazy I was
> hoping somebody had already done it in a shareable form.
>
>
>
> So my question: Does anyone have or know of a shareable integration of
> Apache POI with BaseX for generating Excel spreadsheets or some other
> Excel-generation solution that is not just downloading CSV files?
>
>
>
> Thanks,
>
>
>
> E.
>
> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com <https://www.servicenow.com>
>
> LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter
> <https://twitter.com/servicenow> | YouTube
> <https://www.youtube.com/user/servicenowinc> | Facebook
> <https://www.facebook.com/servicenow>
>

Re: [basex-talk] Possible to Speed Up This Lookup?

[Tim Thompson]

Is there a performance difference when using db:attribute() instead[1]?

Tim

[1] https://docs.basex.org/wiki/Database_Module#db:attribute


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library

On Fri, Jul 15, 2022 at 5:21 PM Eliot Kimber 
wrote:

> Actually, I think I spoke too soon.
>
>
>
> This test:
>
> return (db:option('attrindex'),
>
>   for $node in $map
>
>   return
> prof:track(db:open($rkDatabase)/doc-where-used-index/where-used-entry/@key[xs:integer(.)
> eq 9151416])?time ! util:formatTime(.)
>
>
>
> Returns:
>
>
>
> true
>
> 00.030 seconds
>
>
>
> Where my current database context is the database named in $rkDatabase.
>
>
>
> A small improvement (0.036 to 0.030) but not dramatic.
>
>
>
> So I think I’m still missing some detail, probably a configuration issue.
>
>
>
> Cheers,
>
>
>
> E,
>
> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>
>
>
> *From: *Eliot Kimber 
> *Date: *Friday, July 15, 2022 at 3:39 PM
> *To: *Liam R. E. Quin ,
> basex-talk@mailman.uni-konstanz.de 
> *Subject: *Re: [basex-talk] Possible to Speed Up This Lookup?
>
> Yes, turning on the attribute index definitely improves performa nce
> dramatically, which is no surprise. Not sure how I missed the need to
> expressly turn on these indexes. Doing that now.
>
>
>
> Cheers,
>
>
>
> E.
>
>
>
> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>
>
>
> *From: *Eliot Kimber 
> *Date: *Friday, July 15, 2022 at 3:01 PM
> *To: *Liam R. E. Quin ,
> basex-talk@mailman.uni-konstanz.de 
> *Subject: *Re: [basex-talk] Possible to Speed Up This Lookup?
>
> There are on the order of 50K index entry items.
>
>
>
> I think the attribute index is probably the answer—for some reason I
> assumed that would be indexed by default but I see that it is not looking
> at the database details.
>
>
>
> I’ll try enabling the index and see what happens.
>
>
>
> Cheers,
>
>
>
> E.
>
>
>
> _
>
> *Eliot Kimber*
>
> Sr Staff Content Engineer
>
> O: 512 554 9368
>
> M: 512 554 9368
>
> servicenow.com 
>
> LinkedIn  | Twitter
>  | YouTube
>  | Facebook
> 
>
>
>
> *From: *Liam R. E. Quin 
> *Date: *Friday, July 15, 2022 at 1:22 PM
> *To: *Eliot Kimber ,
> basex-talk@mailman.uni-konstanz.de 
> *Subject: *Re: [basex-talk] Possible to Speed Up This Lookup?
>
> [External Email]
>
>
> On Fri, 2022-07-15 at 18:04 +, Eliot Kimber wrote:
> >  collection($rkDatabase)/doc-where-used-index/where-used-
> > entry[string(@key) eq string(db:node-id($node))]
> >
> > Where the markup is shown in [1] below.
> >
> > Using prof:track() I’m measuring a consistent 0.036 seconds per
> > lookup,
>
> could it be opening the collection that takes time? Or are there
> hundreds of thousands of where-used-entry elements, and, if so, do you
> have an attribute value index?
>
> --
> Liam Quin,
> https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQXLy9rZQ$
> 
> Available for XML/Document/Information Architecture/XSLT/
> XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
> Barefoot Web-slave, antique illustrations:
> https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!C1fj636Ib2y0kENz2AwiUJZj6IyBT5mrb5YE_kSoMeqI0fFs95OpyWpldUalcS47Lg7Dt-1d0gNC7xyQWGj0qnM$
> 
>

Re: [basex-talk] Link Graph Construction: Anything I Can Crib or Learn From?

[Tim Thompson]

Thanks, Bridger. I agree this seems like a use case for graph technologies
(RDF/SPARQL or labeled property graphs). SPARQL 1.1 includes property
paths, which make it possible to query on transitive properties (e.g., A
contains B, B contains C). One example from Wikidata:
https://twitter.com/andre_ourednik/status/1427264453217763336.

There are also document-based representations, as Bridger mentions:
JSON-LD, RDF/XML, and TriX are supported by RDF tools; there's also GraphML
and GEXF, supported by the Gephi platform[1] for network visualization and
also Python tools like NetworkX[2].

Would be interesting to test how a recursive approach using db:attribute,
etc., over a link index would scale in BaseX.

Tim

[1] https://gephi.org/
[2] https://networkx.org/


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library



On Thu, Jun 23, 2022 at 1:09 PM Bridger Dyson-Smith 
wrote:

> Hi Eliot -
>
> I've wondered (but never tested/explored) about leveraging some semblance
> of json-ld (or serialized ttl, or something similar) and passing those
> values to Apache Jena (or another SPARQL processor) to use that as an
> inference engine. I'm deep in Speculation Territory here - I don't know
> what anything would look like - but you're describing an interesting
> problem, and it seems doable. Martynas Jusevicius (and his colleagues) have
> a project, LinkedDataHub, that may provide another avenue for exploring
> this -- I haven't used AtomGraph's applications, but he's active on the
> xml.com slack, and it looks like there are some interesting visualization
> capabilities with their work.
>
> Our listserv friend and neighbor, Tim Thompson of Yale, may have some
> ideas along these lines, too.
> Sorry that I can't provide anything concrete, but I hope some of this is
> somewhat helpful.
> Best,
> Bridger
>
> [1] https://json-ld.org/
> [2] https://jena.apache.org/
> [3] https://github.com/AtomGraph/LinkedDataHub
>
> On Thu, Jun 23, 2022 at 10:35 AM Eliot Kimber 
> wrote:
>
>> In the context of our Project Mirabel system that manages DITA content, I
>> need to be able answer the question “for topic X, what other topics link to
>> it directly or indirectly?”
>>
>>
>>
>> That is, say Topic A links to Topic B that Links to Topic C.
>>
>>
>>
>> Asking the question “What topics ultimately link to topic C?” I would
>> like to get the answer “Topic A, Topic B”.
>>
>>
>>
>> Getting the answer for direct references is easy—I already build a
>> where-used index that captures, for each DITA map or topic, what other maps
>> and topics link directly to it.
>>
>>
>>
>> But to get the Topic A part of the answer I need some kind of link graph
>> index and I’m not sure how best to go about calculating this or capturing
>> it in some index or set of indexes.
>>
>>
>>
>> In our content the fan out from a single Topic C to the set of topics
>> that ultimately reference it could be 10s of 1000s of topics. We have about
>> 45K topics in the content for each version of the ServiceNow Platform and a
>> number of topics that are used by a large number of other topics, so the
>> explosion can be quite large. That suggests that a simple
>> topic-to-ultimately-referenced-topics index would be very inefficient in
>> that the entry for any given topic could potentially have 45K – 1 entries
>> (we don’t care that a topic references itself).
>>
>>
>>
>> On the other hand, working backwards through chains of direct references
>> can also be expensive and is probably too slow, so maybe the brute-force
>> index is the best option?
>>
>>
>>
>> At the same time, I would like to be able to quickly visualize the link
>> graph extending from or ending in any given topic or simply the link graph
>> for the entire information set, which requires capturing the nodes and
>> edges.
>>
>>
>>
>> My question: does anyone either have experience or insight into this kind
>> of link graph challenge or know of relevant papers or general discussion of
>> graph processing I might look at?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Eliot
>>
>> _
>>
>> *Eliot Kimber*
>>
>> Sr Staff Content Engineer
>>
>> O: 512 554 9368
>>
>> M: 512 554 9368
>>
>> servicenow.com <https://www.servicenow.com>
>>
>> LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter
>> <https://twitter.com/servicenow> | YouTube
>> <https://www.youtube.com/user/servicenowinc> | Facebook
>> <https://www.facebook.com/servicenow>
>>
>

[basex-talk] BaseX performance with Apple M1 Max

[Tim Thompson]

Hi, all,

I just had my five-year workstation upgrade for my work laptop. Since BaseX
is my main productivity tool, I was curious to test performance on the new
hardware.

Old machine:
2016 MacBook Pro
2.6 GHz Quad-Core Intel Core i7
16 GB 2133 MHz LPDDR3

New machine:
2021 MacBook Pro
Apple M1 Max
32 GB unified memory

With BaseX 9.7.3 beta, I tested a 4.11 GB XML document that has a
series of *entry
*elements, each representing a name from our library catalog. On the old
machine, it took 9 minutes to create a database from this file, with text
and full-text indexes. On the new machine, it took 3.5 minutes.

I ran the following query on both machines:

{
  for tumbling window $entry in /*/entry
  start $s next $sn when starts-with($sn/key, $s/key)
  end $e next $en when not(starts-with($en/key, $e/key))
  return
{$entry/key}
}

On the old machine, execution time ranged from 40s to 47s. On the new
machine, it ranged from 11.5s to 18s.

In short, it looks as though I'll be a lot more productive now! :)

Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library

Re: [basex-talk] Performance of ft:search function

[Tim Thompson]

Oh, I see--thanks for the tip; I wasn't aware of the SET RUNS feature,
which is really helpful! With 1000 runs, the average execution time is more
in line with expectations: 38.96ms for expression #1 and 12.44ms for #2.
But I notice that with successive executions, #1 gets faster: 38.96ms,
17.73ms, 12.82ms. Is this a result of caching?

Best,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library



On Wed, Apr 27, 2022 at 5:09 PM Christian Grün 
wrote:

> 2. Direct lookup against subindex
>> Time: 3.3ms
>> Expression: ft:search($index, $text)/../..
>>
>> 3. Lookup against subindex file with reference to large index
>> Time: 2.9ms
>> Expression:
>> let $s :=
>>   ft:search($index, $text)/../..
>> return db:open-id($db, $s/id)/../..
>>
>> My question is: why would the third expression be slightly faster (or at
>> least not slower) than the second one, if it involves additional
>> computation?
>>
>
> I assume it's due to slight variations during your measurements. How many
> items will be returned by ft:search? Do you get the same runtime if you run
> the code 100 or 1000 times?
>
> In the GUI, you can type and execute SET RUNS 100 in the top input bar (in
> command mode). Your query will then be executed multiple times, and you
> will get shown the average runtime in the Info View.
>
>
>
>
>

Re: [basex-talk] Stemming in BaseX Full-Text

[Tim Thompson]

Excellent--thanks for this enhancement!

Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library



On Wed, Apr 27, 2022 at 3:43 PM Christian Grün 
wrote:

> > "men" is stemmed to both "man" and "cocksman"--shaking my head on that
> one!
>
>  Reads like a subversive Easter egg :·)
>
> Thanks for the efforts, Tim. The results (memorandum → memo, others)
> indicate to me that a dictionary-based solution has been chosen by
> Bitext.
>
> I have decided to enhance our Porter algorithm with a simple
> dictionary that contains the most common irregular plural forms,
> including the ones you analyzed and various others. A new snapshot is
> available [1,2].
>
> Best,
> Christian
>
> [1] https://github.com/BaseXdb/basex/issues/2097
> [2] https://files.basex.org/releases/latest/
>

[basex-talk] Performance of ft:search function

[Tim Thompson]

Hello,

I have a largish (5.4G) file with a full-text index that I am using to
reconcile names in a local dataset. I've been experimenting with splitting
the file into many smaller index files to improve performance. I group the
entries by initial character and create a new index file for each distinct
initial character. Each smaller file then gets its own full-text index.

I've been following the approach outlined in the documentation for custom
index structures
. Using
prof:track, I've noticed the following performance for different uses of
ft:search.

(Here, $db refers to the 5.4G file, and $index refers to a smaller 159MB
subindex. Times are averaged across 10 runs of 1000 iterations for each
expression.)

1. Direct lookup against large index
Time: 23ms
Expression: ft:search($db, $text)/../..

2. Direct lookup against subindex
Time: 3.3ms
Expression: ft:search($index, $text)/../..

3. Lookup against subindex file with reference to large index
Time: 2.9ms
Expression:
let $s :=
  ft:search($index, $text)/../..
return db:open-id($db, $s/id)/../..

My question is: why would the third expression be slightly faster (or at
least not slower) than the second one, if it involves additional
computation?

Thanks in advance,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library

Re: [basex-talk] Stemming in BaseX Full-Text

[Tim Thompson]

Hi, Christian,

Sorry for being slow to follow up on this. I just ran your list through the
stemming function in ML--results are attached. The original singular and
plural forms are recorded in @a and @b, whereas the corresponding stemmed
forms are recorded in  and .

The results look pretty good to me, with some odd exceptions: e.g., "celli"
is stemmed to "celly," "timpani" is stemmed to "timpani tympanum," and
"men" is stemmed to both "man" and "cocksman"--shaking my head on that one!

All best,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library


On Thu, Apr 14, 2022 at 5:06 AM Christian Grün 
wrote:

> Dear Tim,
>
> It’s exactly how Bridger said: The Porter algorithm is one of the
> fastest available, but it doesn’t provide support for various edge
> cases, including words, however, that are very common, such as
> (wo)?m(a|e)n, child(ren)? or t(oo|ee)th.
>
> We could switch to a more advanced solution. I assume that MarkLogic’s
> Bitext algorithms are only commercially available, but I remember that
> Apache Lucene provides other English stemmers as well, which I haven’t
> tested yet. The vast number of English plural forms is regular, so we
> could also include a dictionary for the most frequent irregular forms.
>
> I have attached a list of frequent English words with irregular plural
> forms to this mail. I would be interested to learn if all of these are
> correctly stemmed by MarkLogic. Would you like to give it a try?
>
> Thanks & cheers,
> Christian
> __
>
> addendum addenda
> alumnus alumni
> analysis analyses
> antithesis antitheses
> apex apices
> appendix appendices
> axis axes
> bacillus bacilli
> bacterium bacteria
> basis bases
> beau beaux
> bureau bureaux
> cactus cacti
> cello celli
> château châteaux
> cherub cherubim
> child children
> codex codices
> concerto concerti
> corpus corpora
> crisis crises
> criterion criteria
> curriculum curricula
> datum data
> diagnosis diagnoses
> die dice
> dwarf dwarves
> ellipsis ellipses
> erratum errata
> fez fezzes
> focus foci
> foot feet
> fungus fungi
> genus genera
> goose geese
> graffito graffiti
> half halves
> hippopotamus hippopotami
> hoof hooves
> hypothesis hypotheses
> index indices
> kibbutz kibbutzim
> lemma lemmata
> libretto libretti
> loaf loaves
> locus loci
> louse lice
> man men
> matrix matrices
> medium media
> memorandum memoranda
> mouse mice
> nucleus nuclei
> oasis oases
> opus opera
> ovum ova
> ox oxen
> parenthesis parentheses
> phenomenon phenomena
> phylum phyla
> polyhedron polyhedra
> quiz quizzes
> radius radii
> referendum referenda
> scarf scarves
> schema schemata
> self selves
> stigma stigmata
> stimulus stimuli
> stratum strata
> syllabus syllabi
> symposium symposia
> synopsis synopses
> tableau tableaux
> thesis theses
> thief thieves
> timpano timpani
> tooth teeth
> uterus uteri
> vertex vertices
> vortex vortices
> wharf wharves
> wife wives
> wolf wolves
> woman women
> __
>
>
> On Thu, Apr 14, 2022 at 12:14 AM Bridger Dyson-Smith
>  wrote:
> >
> > Tim -
> >
> > On Wed, Apr 13, 2022, 4:53 PM Tim Thompson  wrote:
> >>
> >> Thanks, Bridger--that's very helpful! I'm not sure what MarkLogic is
> using exactly, but it seems fairly sophisticated (there's even an advanced
> option for multiple stemming: e.g., "further" has "far," "farther,"
> "further" as stems).
> >
> >
> > Indeed, MarkLogic appears to offer a number of stemmers, some of which
> offer lemmatization functionality. I couldn't say if adding this type of
> capability to BaseX is feasible.
> >>
> >>
> >> All best,
> >> Tim
> >>
> >
> > Best,
> > Bridger
> >
> >>
> >> --
> >> Tim A. Thompson (he, him)
> >> Librarian for Applied Metadata Research
> >> Yale University Library
> >>
> >>
> >>
> >> On Wed, Apr 13, 2022 at 12:13 PM Bridger Dyson-Smith <
> bdysonsm...@gmail.com> wrote:
> >>>
> >>> Hi Tim  -
> >>>
> >>> On Wed, Apr 13, 2022 at 11:40 AM Tim Thompson 
> wrote:
> >>>>
> >>>> I'm currently involved in a project that's using MarkLogic, and I
> noticed that its implementation of English-language stemming differs from
> that of BaseX: e.g., "mouse" and &quo

Re: [basex-talk] Stemming in BaseX Full-Text

[Tim Thompson]

Thanks, Bridger--that's very helpful! I'm not sure what MarkLogic is using
exactly, but it seems fairly sophisticated (there's even an advanced option
for multiple stemming: e.g., "further" has "far," "farther," "further" as
stems).

All best,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library



On Wed, Apr 13, 2022 at 12:13 PM Bridger Dyson-Smith 
wrote:

> Hi Tim  -
>
> On Wed, Apr 13, 2022 at 11:40 AM Tim Thompson  wrote:
>
>> I'm currently involved in a project that's using MarkLogic, and I noticed
>> that its implementation of English-language stemming differs from that of
>> BaseX: e.g., "mouse" and "mice" both stem to "mouse."
>>
>> In BaseX, those words are stemmed separately. Is this a known limitation
>> of the internal English syntax parser?
>>
>> It's my (admittedly, *VERY*) limited understanding that the BaseX
> stemmer, at least for English, is limited to the Porter Stemmer[1]. The
> Porter Stemmer just stems, and doesn't handle stemming from plurals to
> singulars in the case of apophonic plurals.
>
> It'd be interesting to learn what stemmer(s) MarkLogic uses.
>
> And, while I'm not that familiar with it (and it would probably entail
> significant work to implement), the `ft:thesaurus()` function provides
> similar functionality:
> ```
> ft:thesaurus(
>   
> 
>   mice
>   
> mouse
> NT
>   
>   
> rodent
> BTG
>   
> 
>   ,
>   'mice'
> )
> ```
>
>
>> Example:
>>
>> db:create("stem-test",
>>   
>> mouse
>> mice
>>   
>>   , "data", map {"ftindex": true(), "stemming": true(), "language": "en"}
>> )
>> ,
>> update:output(
>>   ft:search("stem-test", "mice")
>> )
>>
>>
>> Thanks,
>> Tim
>>
>>
>>
> Best,
> Bridger
>
> [1]
> https://github.com/BaseXdb/basex/blob/da1e55d0214e44c1532f121c282021db50a9aa51/basex-core/src/main/java/org/basex/util/ft/EnglishStemmer.java
>
>
> --
>> Tim A. Thompson (he, him)
>> Librarian for Applied Metadata Research
>> Yale University Library
>>
>>

[basex-talk] Stemming in BaseX Full-Text

[Tim Thompson]

I'm currently involved in a project that's using MarkLogic, and I noticed
that its implementation of English-language stemming differs from that of
BaseX: e.g., "mouse" and "mice" both stem to "mouse."

In BaseX, those words are stemmed separately. Is this a known limitation of
the internal English syntax parser?

Example:

db:create("stem-test",
  
mouse
mice
  
  , "data", map {"ftindex": true(), "stemming": true(), "language": "en"}
)
,
update:output(
  ft:search("stem-test", "mice")
)


Thanks,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library

Re: [basex-talk] Basexgui Feature suggestion

[Tim Thompson]

Thanks, Christian--somehow I had missed the "Set as context" feature, and
haven't taken advantage of the filter feature before, even though I work a
lot in the BaseX GUI.

One small UI change that would make a big difference (to me) would be to
provide a drop-down option for files in the Editor (à la oXygen XML
Editor), instead of scrolling or stacked tabs. I tend to have many tabs
open at once, and neither of the available options is ideal for switching
back and forth. I should probably just use the project view, but I like to
have more space for the Editor window.

Finally, one small issue using the GUI on a MacBook M1 Pro: when I try to
change the interface language from the Preferences menu, it doesn't take
effect after I restart. I have to manually change the .basex config file to
change the language.

All best,
Tim


-- 
Tim A. Thompson (he, him)
Librarian for Applied Metadata Research
Yale University Library



On Fri, Apr 8, 2022 at 10:20 AM Christian Grün 
wrote:

> Hi Steven,
>
> Thanks to Hans-Jürgen, the "Set as context" menu item was introduced just
> recently.
>
> And thanks to Tamara: Creating intermediate databases is exactly what I
> do, too. If the query output is a valid XML document, it can also be stored
> by clicking on the “Save” icon in the result view panel.
>
> > I get a shorter menu of options that what’s shown in that doc:
>
> The OS shouldn’t matter. I have Windows, but I’ve just been forwarded a
> 9.7 Mac screenshot (attached to this mail), and it looks correct. Can
> anyone else confirm that the menu is not completely shown on a Mac? Do you
> possibly get output some unexpected errors when starting the BaseX GUI from
> command line?
>
> There is another built-in option to bind the result of a query to the
> context. It works if you have a database opened and if the query results
> are nodes from that database. We called the feature “Filter” (the possibly
> suboptimal term is because this feature was added 15 years ago; at that
> time, BaseX was limited to a simple XPath implementation). If you open a
> database and run a query that yields database nodes as results, you can
> press the filter icon: Your next queries will be based on these results,
> and all visualizations will adapt to show the result nodes on top level.
> With the arrow buttons in the menu bar, you can navigate between the
> previous and next result sets, or you can jump one level up (the
> parent::node() step will then be applied to all nodes in the result set) or
> back to the root nodes.
>
> Maybe we can generalize the filter feature to work on arbitrary results,
> no matter what data type they have. One challenge here is that intermediate
> results can get fairly large, so we shouldn’t include them in the history.
> Next, if the results belong to arbitrary database nodes, we’d need to
> create main-memory copies from these nodes or ensure that the context is
> invalidated once a referenced database will be updated or deleted. Next,
> we’ll have to take care of … Alright, I’ll stop ;)
>
> Further suggestions are welcome.
>
> Best,
> Christian
>
>
> > On Apr 7, 2022, at 1:25 PM, Tamara Marnell 
> wrote:
> >
> > Hi Steven,
> >
> > To save a few clicks, you can create a new database directly from your
> results in XQuery, without saving them to a file first. Instead of
> returning the results outright, assign them to a variable to pass as the
> input to db:create() in the return, with a made-up file name for the path.
> >
> > let $results := {
> >   [Your original return in here]
> > }
> > return db:create('my_results_db', $results, 'my_results.xml')
> >
> > Then you can run further queries using db:open('my_results_db')/results,
> and DROP my_results_db in the command input bar when you don't need it
> anymore.
> >
> > This isn't to say a new feature isn't a good idea, just that there's a
> way you can streamline your workflow before that feature exists!
> >
> > -Tamara
> >
> >
> >
> > On Thu, Apr 7, 2022 at 9:49 AM Majewski, Steven Dennis (sdm7g) <
> sd...@virginia.edu> wrote:
> >>
> >>
> >> It would be handy if there was a way to bind results to current context
> for additional inspection/investigation/query of results. Currently, it
> seems, you would have to save results to a file and then create database
> from that file to make an additional query on results.
> >> That binding would be available from editor or input bar.
> >>
> >> Or perhaps alternatively, it could include an option to base
> visualizations on results instead of open database — although I’m guessing
> the former would be easier to implement than the latter.
> >>
> >> — Steve Majewski
>
>

Re: [basex-talk] Keyboard issue on MacBook (Apple M1 Pro)

[Tim Thompson]

Thanks again! I'll switch to ABC - Extended.

Tim

On Mon, Mar 7, 2022 at 4:04 PM Alexander Holupirek 
wrote:

> You're right, Tim, I can not enter the quot/apos as well.
> I think we can file a bug report for this.
>
> Meanwhile it may be an option for you to use "ABC - Extended" as keyboard
> layout.
> It uses the combinations, eg. "OPTION + U + A" for ä and "OPTION + E + E"
> for é.
> Double and single quote works in BaseXGUI as well.
>
> Alex
>
> > On 7. Mar 2022, at 21:27, Tim Thompson  wrote:
> >
> > Thanks, Alex. That's right, the accented character can be entered, but
> the quot/apos key itself can't be entered, so I can type ä, but not "ä". Is
> that the case for you too?
> >
> > Tim
> >
> >
> >
> > On Mon, Mar 7, 2022 at 3:05 PM Alexander Holupirek 
> wrote:
> > > Hello, I'm having a small but annoying issue with keyboard input in
> the BaseX GUI on a new MacBook (M1 Pro). I tend to use the U.S.
> International keyboard layout for ease of typing accented characters: e.g.,
> using the apostrophe/quotation key as a dead key when typing é or ü.
> However, with this keyboard layout, the apos/quot key doesn't register at
> all in the BaseX GUI. It works as expected everywhere else. I've tried
> switching fonts, but that doesn't seem to help. Any ideas? Thanks in
> advance! Tim
> >
> >
> > Hi Tim,
> >
> > I just tried what you described on a M1. Using 'International - PC' as
> keyboard layout and pressing three keys, namely 'SHIFT' + 'QUOT' + 'A'
> gives me the following behaviour:
> >
> > Terminal-App (and other apps):
> >
> > App  | Keys  | Description
> > -|---|---
> > Terminal-App | SHIFT | (nothing happens)
> >  | " | a double quot sign (") in a (yellow)
> highlighted square
> >  | a | the german umlaut a (ä) on default background
> appears
> >
> >
> > In BaseXGUI:
> >
> > App  | Keys  | Description
> > -|---|---
> > BaseXGUI | SHIFT | (nothing happens)
> >  | " | (nothing happens)
> >  | a | the german umlaut a (ä) on default background
> appears
> >
> > So it seems that the indication / user feedback is missing in BaseXGUI,
> but the umlaut character can nevertheless be entered.
> >
> > Do you experience the same?
> > Alex
>
>

Re: [basex-talk] Keyboard issue on MacBook (Apple M1 Pro)

[Tim Thompson]

Thanks, Alex. That's right, the accented character can be entered, but the
quot/apos key itself can't be entered, so I can type ä, but not "ä". Is
that the case for you too?

Tim



On Mon, Mar 7, 2022 at 3:05 PM Alexander Holupirek 
wrote:

> > Hello, I'm having a small but annoying issue with keyboard input in the
> BaseX GUI on a new MacBook (M1 Pro). I tend to use the U.S. International
> keyboard layout for ease of typing accented characters: e.g., using the
> apostrophe/quotation key as a dead key when typing é or ü. However, with
> this keyboard layout, the apos/quot key doesn't register at all in the
> BaseX GUI. It works as expected everywhere else. I've tried switching
> fonts, but that doesn't seem to help. Any ideas? Thanks in advance! Tim
>
>
> Hi Tim,
>
> I just tried what you described on a M1. Using 'International - PC' as
> keyboard layout and pressing three keys, namely 'SHIFT' + 'QUOT' + 'A'
> gives me the following behaviour:
>
> Terminal-App (and other apps):
>
> App  | Keys  | Description
> -|---|---
> Terminal-App | SHIFT | (nothing happens)
>  | " | a double quot sign (") in a (yellow)
> highlighted square
>  | a | the german umlaut a (ä) on default background
> appears
>
>
> In BaseXGUI:
>
> App  | Keys  | Description
> -|---|---
> BaseXGUI | SHIFT | (nothing happens)
>  | " | (nothing happens)
>  | a | the german umlaut a (ä) on default background
> appears
>
> So it seems that the indication / user feedback is missing in BaseXGUI,
> but the umlaut character can nevertheless be entered.
>
> Do you experience the same?
> Alex

[basex-talk] Keyboard issue on MacBook (Apple M1 Pro)

[Tim Thompson]

Hello,

I'm having a small but annoying issue with keyboard input in the BaseX GUI
on a new MacBook (M1 Pro). I tend to use the U.S. International keyboard
layout for ease of typing accented characters: e.g., using the
apostrophe/quotation key as a dead key when typing é or ü. However, with
this keyboard layout, the apos/quot key doesn't register at all in the
BaseX GUI. It works as expected everywhere else. I've tried switching
fonts, but that doesn't seem to help. Any ideas?

Thanks in advance!
Tim



-- 
--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library
344 Winchester Street
New Haven, Connecticut 06511

(203) 432-4189 (office)
(201) 423-9972 (mobile)
www.linkedin.com/in/timathompson
timothy.thomp...@yale.edu

Re: [basex-talk] Issue with HTTP client and authentication

[Tim Thompson]

That solved it! Thank you, Christian, for the outstanding support, as
always. I always panic a bit when I have to think about using something
other than BaseX for my programming needs ;)

Tim



On Wed, Feb 2, 2022 at 6:08 PM Christian Grün 
wrote:

> Thanks.
>
> > The server is MarkLogic, s maybe they are biased against BaseX ;) I'm
> not sure about the configuration, but replacing the User-Agent header with
> "curl/7.71.1" didn't help.
>
> Fraudulent competition ;·)
>
> I’m particularly confused by the response code: 502 usually indicates
> a server-side configuration error, which should not be triggerable
> from a client.
>
> I noticed it would be more correct if BaseX had not sent a second
> request if the response code is unequal to 401. In our case, it might
> have proven to be helpful, as we can see that
>
> a) the response to the second request is 401, and
> b) it contains the Digest WWW-Authenticate header.
>
> In addition, the second request contains a Content-Length header,
> which is missing in the first request. Presumably, due to the missing
> Content-Length:0 header, the server tries to retrieve a body, but
> gives up after the timeout and returns 502.
>
> With the latest snapshot [1], I’m now creating an explicit
> 0-bytes-body for all methods except for GET and TRACE. Does this help?
>
> > I noticed that in both curl and Postman the first request says, "[HTTP
> request 1/2]," whereas BaseX says, "[HTTP request 1/1]"
>
> My hope is that numerous of these irregularities will disappear once
> we’ll have switched to the new JDK 11 HTTP Client [2] with BaseX 10.
>
> [1] https://files.basex.org/releases/latest/
> [2] https://openjdk.java.net/groups/net/httpclient/intro.html
>

Re: [basex-talk] Issue with HTTP client and authentication

[Tim Thompson]

The server is MarkLogic, s maybe they are biased against BaseX ;) I'm not
sure about the configuration, but replacing the User-Agent header with
"curl/7.71.1" didn't help.

Attached is the HTTP info for the curl requests. I also tried it in the
Postman GUI, which worked as well. I noticed that in both curl and Postman
the first request says, "[HTTP request 1/2]," whereas BaseX says, "[HTTP
request 1/1]"

Tim

On Wed, Feb 2, 2022 at 3:38 PM Christian Grün 
wrote:

> Dommage.
>
> Do you know more about the server you are communicating with? Maybe it
> rejects http (POST?) requests sent by Java?
>
> You could try to supply the curl-specific request headers (such as
> User-Agent, and others) with http:send-request.
>
> Could you additionally forward the full Wireshark logs for curl?
>
> I remember that Adam (Retter) added the http-version option to the new
> version of the HTTP Client Module (it’s not available in 1.0 yet). In the
> given case, I’d assume that http-version=1.1 should be okay, because curl
> uses it as well.
>
>
>
> Tim Thompson  schrieb am Mi., 2. Feb. 2022, 21:22:
>
>> Thanks for that. Still no luck, I'm afraid. Attached is the full HTTP
>> info for the requests from BaseX (hostname info replaced with "
>> example.org").
>>
>> I should be able to stick with GET requests for now, but would be nice to
>> have access to POST as well.
>>
>> All best,
>> Tim
>>
>>
>> --
>> Tim A. Thompson
>> Metadata Librarian
>> Yale University Library
>>
>>
>> On Wed, Feb 2, 2022 at 2:20 PM Christian Grün 
>> wrote:
>>
>>> > I notice that curl doesn't include the Authorization header on the
>>> first request, but I am out of my depth here.
>>>
>>> That could be an interesting hint. In the latest snapshot [1], I have
>>> removed the Authorization header from the first request. Can you check
>>> if it makes a difference?
>>>
>>> If not, could you share the headers of the second requests with us?
>>>
>>> [1] https://files.basex.org/releases/latest/
>>>
>>
Hypertext Transfer Protocol
POST /v1/graphs/sparql HTTP/1.1\r\n
[Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n]
Request Method: POST
Request URI: /v1/graphs/sparql
Request Version: HTTP/1.1
Host: example.org\r\n
User-Agent: curl/7.71.1\r\n
Accept: */*\r\n
Content-type: application/sparql-query\r\n
Content-Length: 0\r\n
\r\n
[Full request URI: http://example.org/v1/graphs/sparql]
[HTTP request 1/2]
[Response in frame: 76]
[Next request in frame: 78]


Hypertext Transfer Protocol
HTTP/1.1 401 Unauthorized\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 401 Unauthorized\r\n]
Response Version: HTTP/1.1
Status Code: 401
[Status Code Description: Unauthorized]
Response Phrase: Unauthorized
Date: Wed, 02 Feb 2022 21:11:25 GMT\r\n
Content-Type: application/json; charset=utf-8\r\n
Content-Length: 104\r\n
Connection: keep-alive\r\n
Set-Cookie: 
AWSALB=E+C5iXeqdhPvATdqaa35WllN1c+XOg5C1L0cN04X+K5zzUU9ZYRyociQWkZFyVGksz3M08Y017BV8pFD/llPWEM6FeTqiMoE7q8UJ0E3Fxpk9Rpz5zN6hxl+D/Lp;
 Expires=Wed, 09 Feb 2022 21:11:25 GMT; Path=/\r\n
Set-Cookie: 
AWSALBCORS=E+C5iXeqdhPvATdqaa35WllN1c+XOg5C1L0cN04X+K5zzUU9ZYRyociQWkZFyVGksz3M08Y017BV8pFD/llPWEM6FeTqiMoE7q8UJ0E3Fxpk9Rpz5zN6hxl+D/Lp;
 Expires=Wed, 09 Feb 2022 21:11:25 GMT; Path=/; SameSite=None\r\n
Server: MarkLogic\r\n
WWW-Authenticate: Digest realm="public", qop="auth", 
nonce="3a669a9acad6d8:8nwokPa+YActQ7KDb3R5gg==", opaque="e225558330e43e07"\r\n
\r\n
[HTTP response 1/2]
[Time since request: 0.032887000 seconds]
[Request in frame: 72]
[Next request in frame: 78]
[Next response in frame: 170]
[Request URI: http://example.org/v1/graphs/sparql]
File Data: 104 bytes
JavaScript Object Notation: application/json


Hypertext Transfer Protocol
POST /v1/graphs/sparql HTTP/1.1\r\n
[Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n]
Request Method: POST
Request URI: /v1/graphs/sparql
Request Version: HTTP/1.1
Host: example.org\r\n
 [truncated]Authorization: Digest username="[...]", realm="public", 
nonce="[...]", uri="/v1/graphs/sparql", cnonce="[...]", nc=0001, qop=auth, 
response="[...]"
User-Agent: curl/7.71.1\r\n
Accept: */*\r\n
Content-type: application/sparql-query\r\n
Content-Length: 35\r\n
\r\n
[Full request URI: http://example.org/v1/graphs/sparql]
[HTTP request 2/2]
[Prev request in frame: 72]
[Response in frame: 170]
   

Re: [basex-talk] Issue with HTTP client and authentication

[Tim Thompson]

Thanks, Ash! That attribute doesn't seem to be available in BaseX, as far
as I can tell.

--
Tim A. Thompson
Metadata Librarian
Yale University Library


On Wed, Feb 2, 2022 at 2:21 PM Clark, Ash  wrote:

> In case it helps: I had a similar problem in eXist some years ago. My fix
> was to put the attribute `http-version="1.0"` on the request element. The
> HTTP client defaults to HTTP v1.1, but for whatever reason, only HTTP 1.0
> worked for me.
>
> ~Ash
> --
> *From:* BaseX-Talk  on behalf
> of Tim Thompson 
> *Sent:* Wednesday, February 2, 2022 1:59 PM
> *To:* Christian Grün ; BaseX <
> basex-talk@mailman.uni-konstanz.de>
> *Subject:* Re: [basex-talk] Issue with HTTP client and authentication
>
> Ah! Thanks for clarifying that. So, now it is working for GET requests in
> BaseX, but still not for POST.
>
> So, this works:
> http:send-request(
>  method="GET"
>   href="{$endpoint}?query={encode-for-uri('select * where {?s ?p ?o}
> limit 1')}"
>   username="{$user}"
>   password="{$pass}"
>   auth-method="Digest">
>   
> )
>
> But when I try to POST a request body with Digest auth, it is still
> failing (httpbin doesn't seem to support testing Digest with POST,
> unfortunately). The request hangs for a while, then fails with 401
> Unauthorized. Wireshark reveals that the first network response from the
> server is actually a 502 Bad Gateway.
>
> Attached are two screenshots from Wireshark: first, the request/response
> sequence with curl, then with BaseX.
>
> Here's what Wireshark shows for the first request via curl:
>
> Hypertext Transfer Protocol
> POST /v1/graphs/sparql HTTP/1.1\r\n
> [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n]
> [POST /v1/graphs/sparql HTTP/1.1\r\n]
> [Severity level: Chat]
> [Group: Sequence]
> Request Method: POST
> Request URI: /v1/graphs/sparql
> Request Version: HTTP/1.1
> Host: example.org
> <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D=0>
> \r\n
> User-Agent: curl/7.71.1\r\n
> Accept: */*\r\n
> Content-type: application/sparql-query\r\n
> Content-Length: 0\r\n
> [Content length: 0]
> \r\n
> [Full request URI: example.org/v1/graphs/sparql
> <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D=0>
> ]
>
> Then via BaseX:
>
> Hypertext Transfer Protocol
> POST /v1/graphs/sparql HTTP/1.1\r\n
> [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n]
> [POST /v1/graphs/sparql HTTP/1.1\r\n]
> [Severity level: Chat]
> [Group: Sequence]
> Request Method: POST
> Request URI: /v1/graphs/sparql
> Request Version: HTTP/1.1
> Content-type: application/sparql-query\r\n
> Authorization: Digest\r\n
> User-Agent: Java/1.8.0_121\r\n
> Host: example.org
> <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D=0>
> \r\n
> Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n
> Connection: keep-alive\r\n
> \r\n
> [Full request URI: example.org/v1/graphs/sparql
> <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D=0>
> ]
>
> I notice that curl doesn't include the Authorization header on the first
> request, but I am out of my depth here.
>
> Best,
> Tim
>
>

Re: [basex-talk] Issue with HTTP client and authentication

[Tim Thompson]

Thanks for that. Still no luck, I'm afraid. Attached is the full HTTP info
for the requests from BaseX (hostname info replaced with "example.org").

I should be able to stick with GET requests for now, but would be nice to
have access to POST as well.

All best,
Tim


--
Tim A. Thompson
Metadata Librarian
Yale University Library


On Wed, Feb 2, 2022 at 2:20 PM Christian Grün 
wrote:

> > I notice that curl doesn't include the Authorization header on the first
> request, but I am out of my depth here.
>
> That could be an interesting hint. In the latest snapshot [1], I have
> removed the Authorization header from the first request. Can you check
> if it makes a difference?
>
> If not, could you share the headers of the second requests with us?
>
> [1] https://files.basex.org/releases/latest/
>

Hypertext Transfer Protocol
POST /v1/graphs/sparql HTTP/1.1\r\n
[Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n]
Request Method: POST
Request URI: /v1/graphs/sparql
Request Version: HTTP/1.1
Content-Type: application/sparql-query\r\n
User-Agent: Java/1.8.0_121\r\n
Host: example.org\r\n
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n
Connection: keep-alive\r\n
\r\n
[Full request URI: http://example.org/v1/graphs/sparql]
[HTTP request 1/1]
[Response in frame: 1996]


Hypertext Transfer Protocol
HTTP/1.1 502 Bad Gateway\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 502 Bad Gateway\r\n]
Response Version: HTTP/1.1
Status Code: 502
[Status Code Description: Bad Gateway]
Response Phrase: Bad Gateway
Server: awselb/2.0\r\n
Date: Wed, 02 Feb 2022 20:18:53 GMT\r\n
Content-Type: text/html\r\n
Content-Length: 122\r\n
Connection: keep-alive\r\n
Set-Cookie: 
AWSALB=lF1WLH+hKED77jWmN61V9QdOTP0BJ1fXpu8LHiDxZm9jHPQvyexXDCsnhXBpYue4LwcWWu2lewfg3bA12h8IKWFcYaX4rssqpdOn6thj5uwnvLROFVJzcq/hKDUa;
 Expires=Wed, 09 Feb 2022 20:18:22 GMT; Path=/\r\n
Set-Cookie: 
AWSALBCORS=lF1WLH+hKED77jWmN61V9QdOTP0BJ1fXpu8LHiDxZm9jHPQvyexXDCsnhXBpYue4LwcWWu2lewfg3bA12h8IKWFcYaX4rssqpdOn6thj5uwnvLROFVJzcq/hKDUa;
 Expires=Wed, 09 Feb 2022 20:18:22 GMT; Path=/; SameSite=None\r\n
\r\n
[HTTP response 1/1]
[Time since request: 30.071527000 seconds]
[Request in frame: 290]
[Request URI: http://example.org/v1/graphs/sparql]
File Data: 122 bytes
Line-based text data: text/html (6 lines)


Hypertext Transfer Protocol
POST /v1/graphs/sparql HTTP/1.1\r\n
[Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n]
Request Method: POST
Request URI: /v1/graphs/sparql
Request Version: HTTP/1.1
Content-Type: application/sparql-query\r\n
User-Agent: Java/1.8.0_121\r\n
Host: example.org\r\n
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n
Connection: keep-alive\r\n
Content-Length: 75\r\n
\r\n
[Full request URI: http://example.org/v1/graphs/sparql]
[HTTP request 1/1]
[Response in frame: 2009]
File Data: 75 bytes
Media Type
Media type: application/sparql-query (75 bytes)


Hypertext Transfer Protocol
HTTP/1.1 401 Unauthorized\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 401 Unauthorized\r\n]
Response Version: HTTP/1.1
Status Code: 401
[Status Code Description: Unauthorized]
Response Phrase: Unauthorized
Date: Wed, 02 Feb 2022 20:18:53 GMT\r\n
Content-Type: text/html; charset=utf-8\r\n
Content-Length: 209\r\n
Connection: keep-alive\r\n
Set-Cookie: 
AWSALB=ryXq9S0wuk8w1bZpm1JfHWiQPQskZ0578vVFTt8u/P+v8PDFsqZmA4dD1FbWEyCjxCPVybaNkVozB9lgjl0a9VoS/YZeoxphyRCkDCRNpmSedULCBp178yBMeSp7;
 Expires=Wed, 09 Feb 2022 20:18:53 GMT; Path=/\r\n
Set-Cookie: 
AWSALBCORS=ryXq9S0wuk8w1bZpm1JfHWiQPQskZ0578vVFTt8u/P+v8PDFsqZmA4dD1FbWEyCjxCPVybaNkVozB9lgjl0a9VoS/YZeoxphyRCkDCRNpmSedULCBp178yBMeSp7;
 Expires=Wed, 09 Feb 2022 20:18:53 GMT; Path=/; SameSite=None\r\n
Server: MarkLogic\r\n
WWW-Authenticate: Digest realm="public", qop="auth", 
nonce="3a669343b0a58c:Pr0WdWn3yn7M6bEt1ZAslw==", opaque="13d146b5e7f6c182"\r\n
\r\n
[HTTP response 1/1]
[Time since request: 0.042191000 seconds]
[Request in frame: 2006]
[Request URI: http://example.org/v1/graphs/sparql]
File Data: 209 bytes
Line-based text data: text/html (9 lines)

Re: [basex-talk] Issue with HTTP client and authentication

[Tim Thompson]

Thanks, Andy! That's very handy. So, when I test against httpbin, the same
thing happens.

curl request:
curl --digest --user user:pw -X GET '
http://httpbin.org/digest-auth/auth/user/pw'

response:
{
  "authenticated": true,
  "user": "user"
}

BaseX request:
let $endpoint := "http://httpbin.org/digest-auth/auth/user/pw;

let $response := (
  http:send-request(



  )
)
return
  $response

response:
http://expath.org/ns/http-client; status="401"
message="UNAUTHORIZED">
  
  
  
  
  
  
  
  
  
  



--
Tim A. Thompson
Metadata Librarian
Yale University Library


On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce  wrote:

> I find  httpbin.org a useful resource to test this kind of thing [1] and
> [2]
> /Andy
>
> [1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_
> [2] http://httpbin.org/basic-auth/user1/mypass
>
> On Wed, 2 Feb 2022 at 16:58, Christian Grün 
> wrote:
>
>> Good to hear; so at least Digest is detected at this stage ;)
>>
>>
>> Tim Thompson  schrieb am Mi., 2. Feb. 2022, 17:54:
>>
>>> With "digest" I get: "[experr:HC0004] Invalid authentication method:
>>> digest."
>>>
>>> --
>>> Tim A. Thompson
>>> Metadata Librarian
>>> Yale University Library
>>>
>>> On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <
>>> christian.gr...@gmail.com> wrote:
>>>
>>>> Yes, it should still be supported. I assume it doesn't make a
>>>> difference if you use "Digest" or "digest"?
>>>>
>>>>
>>>>
>>>> Tim Thompson  schrieb am Mi., 2. Feb. 2022, 17:48:
>>>>
>>>>> Thanks, Christian. Right, that wasn't a working example. The server
>>>>> I'm querying is behind a firewall, so I can't really reproduce the issue
>>>>> unfortunately.
>>>>>
>>>>> The BaseX documentation states that the HTTP client does support
>>>>> Digest authentication. However, I did some poking around using Wireshark
>>>>> and limited knowledge of network traffic. When I run the request in curl,
>>>>> the expected protocol is played out, and I see the evidence in Wireshark:
>>>>>
>>>>> (1) The server responds with "401 Unauthorized" and provides the nonce
>>>>> value.
>>>>> (2) The client (curl) then does its business (supplies the username
>>>>> and password, resends the request, etc.)
>>>>>
>>>>> But when I submit the request via BaseX, it never gets past step (1).
>>>>> It never seems to send the username and password at all. Is Digest
>>>>> authentication still supported?
>>>>>
>>>>> Best,
>>>>> Tim
>>>>>
>>>>>
>>>>> --
>>>>> Tim A. Thompson
>>>>> Metadata Librarian
>>>>> Yale University Library
>>>>>
>>>>>
>>>>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <
>>>>> christian.gr...@gmail.com> wrote:
>>>>>
>>>>>> Hi Tim,
>>>>>>
>>>>>> Difficult to tell; both the curl and the XQuery variants give me 404.
>>>>>> Do you think you could provide us with an example that works out of
>>>>>> the box?
>>>>>>
>>>>>> Best,
>>>>>> Christian
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson 
>>>>>> wrote:
>>>>>> >
>>>>>> > Hello,
>>>>>> >
>>>>>> > I'm trying to post a SPARQL query to an endpoint using Digest
>>>>>> authentication with the HTTP client.
>>>>>> >
>>>>>> > The query works fine using curl:
>>>>>> >
>>>>>> > curl --digest --user user:pass -X POST -d@'test.rq' \
>>>>>> > -H "Content-type: application/sparql-query" \
>>>>>> > 'http://example.org/sparql'
>>>>>> >
>>>>>> > But the equivalent request in BaseX fails with 401 Unauthorized:
>>>>>> >
>>>>>> > let $endpoint := "http://example.org/sparql;
>>>>>> > let $user := "user"
>>>>>> > let $pass := "pass"
>>>>>> > let $type := "application/sparql-query"
>>>>>> >
>>>>>> > let $response := (
>>>>>> >   http:send-request(
>>>>>> > >>>>> > method="POST"
>>>>>> > href="{$endpoint}"
>>>>>> > username="{$user}"
>>>>>> > password="{$pass}"
>>>>>> > auth-method="Digest"
>>>>>> > send-authorization="true">
>>>>>> > >>>>> > name="Content-Type"
>>>>>> > value="{$type}; charset=utf-8"/>
>>>>>> > >>>>> > media-type="{$type}">{
>>>>>> > ``[
>>>>>> >   select * where {?s ?p ?o} limit 1
>>>>>> > ]``
>>>>>> >   }
>>>>>> > 
>>>>>> >   )
>>>>>> > )
>>>>>> > return
>>>>>> >   $response
>>>>>> >
>>>>>> > Any ideas about what might be causing the BaseX HTTP client to be
>>>>>> denied here?
>>>>>> >
>>>>>> > Thanks in advance,
>>>>>> > Tim
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Tim A. Thompson
>>>>>> > Metadata Librarian
>>>>>> > Yale University Library
>>>>>> >
>>>>>>
>>>>>

Re: [basex-talk] Issue with HTTP client and authentication

[Tim Thompson]

With "digest" I get: "[experr:HC0004] Invalid authentication method:
digest."

--
Tim A. Thompson
Metadata Librarian
Yale University Library

On Wed, Feb 2, 2022 at 11:52 AM Christian Grün 
wrote:

> Yes, it should still be supported. I assume it doesn't make a difference
> if you use "Digest" or "digest"?
>
>
>
> Tim Thompson  schrieb am Mi., 2. Feb. 2022, 17:48:
>
>> Thanks, Christian. Right, that wasn't a working example. The server I'm
>> querying is behind a firewall, so I can't really reproduce the issue
>> unfortunately.
>>
>> The BaseX documentation states that the HTTP client does support Digest
>> authentication. However, I did some poking around using Wireshark and
>> limited knowledge of network traffic. When I run the request in curl, the
>> expected protocol is played out, and I see the evidence in Wireshark:
>>
>> (1) The server responds with "401 Unauthorized" and provides the nonce
>> value.
>> (2) The client (curl) then does its business (supplies the username and
>> password, resends the request, etc.)
>>
>> But when I submit the request via BaseX, it never gets past step (1). It
>> never seems to send the username and password at all. Is Digest
>> authentication still supported?
>>
>> Best,
>> Tim
>>
>>
>> --
>> Tim A. Thompson
>> Metadata Librarian
>> Yale University Library
>>
>>
>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün 
>> wrote:
>>
>>> Hi Tim,
>>>
>>> Difficult to tell; both the curl and the XQuery variants give me 404.
>>> Do you think you could provide us with an example that works out of
>>> the box?
>>>
>>> Best,
>>> Christian
>>>
>>>
>>>
>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson  wrote:
>>> >
>>> > Hello,
>>> >
>>> > I'm trying to post a SPARQL query to an endpoint using Digest
>>> authentication with the HTTP client.
>>> >
>>> > The query works fine using curl:
>>> >
>>> > curl --digest --user user:pass -X POST -d@'test.rq' \
>>> > -H "Content-type: application/sparql-query" \
>>> > 'http://example.org/sparql'
>>> >
>>> > But the equivalent request in BaseX fails with 401 Unauthorized:
>>> >
>>> > let $endpoint := "http://example.org/sparql;
>>> > let $user := "user"
>>> > let $pass := "pass"
>>> > let $type := "application/sparql-query"
>>> >
>>> > let $response := (
>>> >   http:send-request(
>>> > >> > method="POST"
>>> > href="{$endpoint}"
>>> > username="{$user}"
>>> > password="{$pass}"
>>> > auth-method="Digest"
>>> > send-authorization="true">
>>> > >> > name="Content-Type"
>>> > value="{$type}; charset=utf-8"/>
>>> > >> > media-type="{$type}">{
>>> > ``[
>>> >   select * where {?s ?p ?o} limit 1
>>> > ]``
>>> >   }
>>> > 
>>> >   )
>>> > )
>>> > return
>>> >   $response
>>> >
>>> > Any ideas about what might be causing the BaseX HTTP client to be
>>> denied here?
>>> >
>>> > Thanks in advance,
>>> > Tim
>>> >
>>> >
>>> > --
>>> > Tim A. Thompson
>>> > Metadata Librarian
>>> > Yale University Library
>>> >
>>>
>>

Re: [basex-talk] Issue with HTTP client and authentication

[Tim Thompson]

Thanks, Christian. Right, that wasn't a working example. The server I'm
querying is behind a firewall, so I can't really reproduce the issue
unfortunately.

The BaseX documentation states that the HTTP client does support Digest
authentication. However, I did some poking around using Wireshark and
limited knowledge of network traffic. When I run the request in curl, the
expected protocol is played out, and I see the evidence in Wireshark:

(1) The server responds with "401 Unauthorized" and provides the nonce
value.
(2) The client (curl) then does its business (supplies the username and
password, resends the request, etc.)

But when I submit the request via BaseX, it never gets past step (1). It
never seems to send the username and password at all. Is Digest
authentication still supported?

Best,
Tim


--
Tim A. Thompson
Metadata Librarian
Yale University Library


On Wed, Feb 2, 2022 at 10:13 AM Christian Grün 
wrote:

> Hi Tim,
>
> Difficult to tell; both the curl and the XQuery variants give me 404.
> Do you think you could provide us with an example that works out of
> the box?
>
> Best,
> Christian
>
>
>
> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson  wrote:
> >
> > Hello,
> >
> > I'm trying to post a SPARQL query to an endpoint using Digest
> authentication with the HTTP client.
> >
> > The query works fine using curl:
> >
> > curl --digest --user user:pass -X POST -d@'test.rq' \
> > -H "Content-type: application/sparql-query" \
> > 'http://example.org/sparql'
> >
> > But the equivalent request in BaseX fails with 401 Unauthorized:
> >
> > let $endpoint := "http://example.org/sparql;
> > let $user := "user"
> > let $pass := "pass"
> > let $type := "application/sparql-query"
> >
> > let $response := (
> >   http:send-request(
> >  > method="POST"
> > href="{$endpoint}"
> > username="{$user}"
> > password="{$pass}"
> > auth-method="Digest"
> > send-authorization="true">
> >  > name="Content-Type"
> > value="{$type}; charset=utf-8"/>
> >  > media-type="{$type}">{
> > ``[
> >   select * where {?s ?p ?o} limit 1
> > ]``
> >   }
> > 
> >   )
> > )
> > return
> >   $response
> >
> > Any ideas about what might be causing the BaseX HTTP client to be denied
> here?
> >
> > Thanks in advance,
> > Tim
> >
> >
> > --
> > Tim A. Thompson
> > Metadata Librarian
> > Yale University Library
> >
>

[basex-talk] Issue with HTTP client and authentication

[Tim Thompson]

Hello,

I'm trying to post a SPARQL query to an endpoint using Digest
authentication with the HTTP client.

The query works fine using curl:

curl --digest --user user:pass -X POST -d@'test.rq' \
-H "Content-type: application/sparql-query" \
'http://example.org/sparql'

But the equivalent request in BaseX fails with 401 Unauthorized:

let $endpoint := "http://example.org/sparql;
let $user := "user"
let $pass := "pass"
let $type := "application/sparql-query"

let $response := (
  http:send-request(


{
``[
  select * where {?s ?p ?o} limit 1
]``
  }

  )
)
return
  $response

Any ideas about what might be causing the BaseX HTTP client to be denied
here?

Thanks in advance,
Tim


--
Tim A. Thompson
Metadata Librarian
Yale University Library

Re: [basex-talk] Question about ft:normalize

[Tim Thompson]

Thanks again.

Does BaseX support any Unicode block properties, such as
\p{InCombiningDiacriticalMarks}, in regex functions? \p{Mn} works, but
\p{InCombiningDiacriticalMarks} doesn't seem to.

Tim


--
Tim A. Thompson (*he, him*)
Librarian for Applied Metadata Research
Yale University Library
www.linkedin.com/in/timathompson
timothy.thomp...@yale.edu


On Tue, Nov 23, 2021 at 11:16 AM Christian Grün 
wrote:

> It’s US-ASCII (7 bit).
>
>
> Tim Thompson  schrieb am Di., 23. Nov. 2021, 17:07:
>
>> Thanks, Christian. What is the effective character set used when
>> diacritics are removed? Latin-1?
>>
>> Tim
>>
>>
>> --
>> Tim A. Thompson (*he, him*)
>> Librarian for Applied Metadata Research
>> Yale University Library
>> www.linkedin.com/in/timathompson
>> timothy.thomp...@yale.edu
>>
>> On Mon, Nov 22, 2021 at 2:53 PM Christian Grün 
>> wrote:
>>
>>> Hi Tim,
>>>
>>> > I have a question about the BaseX ft:normalize function. What kind of
>>> Unicode normalization is performed by this function, and how might it be
>>> implemented using standard XPath functions?
>>>
>>> The function is based on a custom BaseX tokenization, which includes
>>> normalization of case, removal of diacritics and (if enabled)
>>> language-based stemming. It would be rather challenging to implement
>>> the behavior with standard XPath (that’s mostly why we introduced
>>> ft:tokenize and ft:normalize). If you are looking for a starting
>>> point, you could begin with the FtTokenize Java class [1].
>>>
>>> Hope this helps,
>>> Christian
>>>
>>> [1]
>>> https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/query/func/ft/FtTokenize.java#L31-L51
>>>
>>

Re: [basex-talk] Question about ft:normalize

[Tim Thompson]

Thanks, Christian. What is the effective character set used when diacritics
are removed? Latin-1?

Tim


--
Tim A. Thompson (*he, him*)
Librarian for Applied Metadata Research
Yale University Library
www.linkedin.com/in/timathompson
timothy.thomp...@yale.edu

On Mon, Nov 22, 2021 at 2:53 PM Christian Grün 
wrote:

> Hi Tim,
>
> > I have a question about the BaseX ft:normalize function. What kind of
> Unicode normalization is performed by this function, and how might it be
> implemented using standard XPath functions?
>
> The function is based on a custom BaseX tokenization, which includes
> normalization of case, removal of diacritics and (if enabled)
> language-based stemming. It would be rather challenging to implement
> the behavior with standard XPath (that’s mostly why we introduced
> ft:tokenize and ft:normalize). If you are looking for a starting
> point, you could begin with the FtTokenize Java class [1].
>
> Hope this helps,
> Christian
>
> [1]
> https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/query/func/ft/FtTokenize.java#L31-L51
>

[basex-talk] Question about ft:normalize

[Tim Thompson]

Hello,

I have a question about the BaseX ft:normalize function. What kind of
Unicode normalization is performed by this function, and how might it be
implemented using standard XPath functions?

Thanks in advance!
Tim


--
Tim A. Thompson (*he, him*)
Librarian for Applied Metadata Research
Yale University Library
www.linkedin.com/in/timathompson
timothy.thomp...@yale.edu

[basex-talk] Fuzzy full-text query

[Tim Thompson]

Hello,

I am trying to match a local dataset against entries in a full-text index.
I ran into the following case and was confused about the fuzzy querying
performance:

Baxter, Richard, 1615-1691[text() contains text "Baxter, Richard,
615-1691" using fuzzy]

The BaseX documentation says that the default fuzzy search will preserve a
minimum of 1 error. However, the query above returns no result. If I add
the "errors" parameter with a value of 1, it does work:

Baxter, Richard, 1615-1691[text() contains text "Baxter, Richard,
615-1691" using fuzzy 1 errors]

Is this expected behavior?

Thanks in advance,
Tim


--
Tim A. Thompson
Librarian for Applied Metadata Research
Yale University Library

Re: [basex-talk] Error with full-text and fuzzy query

[Tim Thompson]

Thanks, Christian. The rest of the stack trace was just many lines of "at
org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74)" until
the end:

[qtp1546693040-47] WARN org.eclipse.jetty.server.handler.ErrorHandler -
Error page too large: 500 java.lang.StackOverflowError Request(POST //
10.5.157.229:10214/dba/query-eval)@7dcadc62
[qtp1546693040-47] INFO org.eclipse.jetty.server.handler.ErrorHandler -
Disabling showsStacks for ErrorPageErrorHandler@53032c30{STARTED}

Best regards,
Tim


--
Tim A. Thompson
Metadata Librarian
Yale University Library

On Fri, Jun 4, 2021 at 1:30 PM Christian Grün 
wrote:

> Hi Tim,
>
> > [qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel -
> /dba/query-eval
> > java.lang.StackOverflowError
> > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73)
> > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74)
>
> Could you share some more lines of the stack trace with us?
>
> Thanks in advance
> Christian
>

[basex-talk] Error with full-text and fuzzy query

[Tim Thompson]

Hello,

I have a full-text index defined on a database with 21,936,670 entries. I'm
hitting a stack overflow error when I try to query the following
(normalized) string with a fuzzy query:

"sajawandi, siraj al-din muhammad ibn muhammad, active 12th century.
faraʼid al-sirajiyah"

The index entries look like this:

http://id.loc.gov/authorities/names/n2008037847;>
  Sajāwandī, Sirāj al-Dīn Muḥammad ibn Muḥammad, active 12th century.
Farāʼiḍ al-Sirājīyah
  12th cent.


If I create a new DB ("ftindex-test") with only that one entry and try the
following lookup, it works:

ft:search(
  "ftindex-test",
  "sajawandi, siraj al-din muhammad ibn muhammad, active 12th century.
faraʼid al-sirajiyah"
  , map {"mode": "phrase", "content": "entire", "fuzzy": true()}
)/..

However, if I try the same query against the full index (with ~22 million
entries), it fails with a stack overflow error:

[qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel -
/dba/query-eval
java.lang.StackOverflowError
at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73)
at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74)
...

If I remove the "fuzzy" parameter, the query does work against the full
index. Is this a bug, a known limitation here, or something that I'm
missing?

Thanks in advance,
Tim


--
Tim A. Thompson
Metadata Librarian
Yale University Library

Re: [basex-talk] BaseX with Apache

[Tim Thompson]

Thanks, Bridger! Adding that param seemed to do the trick!

Best,
Tim


On Thu, May 27, 2021 at 9:59 AM Bridger Dyson-Smith 
wrote:

> Hi Tim -
>
> What about modifying the BaseX web.xml in $basex/webapp/WEB-INF? That was
> you'd be disabling in Jetty instead of Apache. To be honest I have no idea
> if this will work, but:
>
> 
>   default
>   org.eclipse.jetty.servlet.DefaultServlet
>   
> dirAllowed
> false
>   
> 
>  or some variation thereof? I cribbed this from stackoverflow, naturally
> (and here's where I found it: [1])
>
> Hope that's helpful, apologies if it's only noise.
> Best,
> Bridger
>
> [1]
> https://stackoverflow.com/questions/7217969/how-to-disable-directory-listing-for-jettys-webappcontext
>
> On Thu, May 27, 2021 at 9:48 AM Tim Thompson  wrote:
>
>> With apologies for the not-strictly-BaseX question--but I'm running BaseX
>> through a reverse proxy with Apache (and Let's Encrypt), and I can't figure
>> out how to disable directory listing for the BaseX webapp/static directory.
>> I've tried editing the apache2 conf files in the standard ways. Hoping
>> someone here has a quick solution to share.
>>
>> Thanks in advance,
>> Tim
>>
>> --
>> Tim A. Thompson
>> Metadata Librarian
>> Yale University Library
>>
>>

[basex-talk] BaseX with Apache

[Tim Thompson]

With apologies for the not-strictly-BaseX question--but I'm running BaseX
through a reverse proxy with Apache (and Let's Encrypt), and I can't figure
out how to disable directory listing for the BaseX webapp/static directory.
I've tried editing the apache2 conf files in the standard ways. Hoping
someone here has a quick solution to share.

Thanks in advance,
Tim

--
Tim A. Thompson
Metadata Librarian
Yale University Library

Re: [basex-talk] Jobs and file modules

[Tim Thompson]

Thanks. I'm still trying to get this to work. Is it possible to put
updating expressions in a library module function (with the name of the
database hard coded) and then call from the function within jobs:eval() in
a main module? When I do this, the jobs don't seem to run in parallel. But
if I put the updating expressions in the main module, the jobs do seem to
run in parallel. Is this a limitation?

I have millions of updates (inserts) that I'm trying to run on 10 large
databases (5GB each). In my current process, it takes about 48 hours to
update a single DB. Are there other options you'd recommend in order to
speed things up?

All best,
Tim


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

On Wed, Feb 10, 2021 at 3:27 AM Christian Grün 
wrote:

> Hi Tim,
>
> Updates can be run in parallel if the name of the database is directly
> specified in the query [1]:
>
>   jobs:eval('delete node db:open("db1")//abc'),
>   jobs:eval('delete node db:open("db2")//def')
>
> In a future version of BaseX, we might split up our compilation phase
> into multiple ones. After this, we could statically detect that a
> passed on variable will be the name of a database.
>
> Until then, you could try to build a query string that included
> hard-coded database names.
>
> Hope this helps,
> Christian
>
> [1] https://docs.basex.org/wiki/Transaction_Management#XQuery
>
>
>
> On Wed, Feb 10, 2021 at 1:56 AM Tim Thompson  wrote:
> >
> > Thank you, Christian, for the detailed explanation!
> >
> > One more question, if I may. Is it possible to run updating jobs on
> different databases in parallel? Or can database update operations only be
> run sequentially, one db at a time? I have a query that calls a function to
> perform a series of operations:
> >
> > for $i in (0 to 9)
> > return (
> >   jobs:eval('
> > declare variable $iter external;
> > local:add-uris("marc.exp.20210115."||$iter)
> >   ', map {"iter": $i})
> > )
> >
> > The function:
> >
> > opens a database
> > iterates through its records
> > performs lookups against an index
> > inserts any matches into the database
> > calls file:append-text-lines() to write the results of the lookups
> >
> > Based on some simple tests, it doesn't seem possible to run the jobs in
> parallel, but I thought I would ask--to see whether there was something I
> was missing.
> >
> > Thanks again,
> > Tim
>

Re: [basex-talk] Jobs and file modules

[Tim Thompson]

Thank you, Christian, for the detailed explanation!

One more question, if I may. Is it possible to run updating jobs on
different databases in parallel? Or can database update operations only be
run sequentially, one db at a time? I have a query that calls a function to
perform a series of operations:

for $i in (0 to 9)
return (
  jobs:eval('
declare variable $iter external;
local:add-uris("marc.exp.20210115."||$iter)
  ', map {"iter": $i})
)

The function:

   - opens a database
   - iterates through its records
   - performs lookups against an index
   - inserts any matches into the database
   - calls file:append-text-lines() to write the results of the lookups

Based on some simple tests, it doesn't seem possible to run the jobs in
parallel, but I thought I would ask--to see whether there was something I
was missing.

Thanks again,
Tim


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

On Sat, Feb 6, 2021 at 5:22 PM Christian Grün 
wrote:

> Hi Tim,
>
> file:write uses the default W3 serialization method "XML". This means
> that the standard entities (&, <, etc.) will be encoded. This can be
> circumvented by using the 'text' output method…
>
>   file:write(..., ..., map { 'method': 'text' })
>
> …or file:write-text.
>
> In BaseX, we introduced our own serialization method 'basex', which
> serializes strings as strings and basex64 and hex data as bytes. With
> this method (if it had been part of the official standard),
> file:write-text and file:write-binary could actually have been
> dropped.
>
> > Ah, never mind. When I run the file:write-text() without jobs:eval(), I
> get an error, "Cannot convert xs:dateTime to xs:string." Is it possible to
> return the error from a job call?
>
> You can cache the result of a query…
>
>   let $job-id := jobs:eval(..., ..., map { 'cache': true() })
>
> …and retrieve the result or the error with jobs:result($job-id).
>
> Hope this helps,
> Christian
>

Re: [basex-talk] Jobs and file modules

[Tim Thompson]

Ah, never mind. When I run the *file:write-text()* without* jobs:eval()*, I
get an error, "Cannot convert xs:dateTime to xs:string." Is it possible to
return the error from a job call?

TIm

--
Tim A. Thompson
Metadata Librarian
Yale University Library

On Sat, Feb 6, 2021 at 10:00 AM Tim Thompson  wrote:

> Hello,
>
> I was experimenting with the jobs module and wondering why there's a
> difference between *file:write() *and *file:write-text()* in the query
> below:
>
> for $i in 1 to 5
> return (
>   jobs:eval('
> declare variable $iter external;
> file:write-text("~/Desktop/job"|| $iter ||".txt", (prof:sleep(5000),
> string(current-dateTime(
> ', map {"iter": $i}
>   )
> )
>
> With *file:write-text()*, if I don't wrap *current-dateTime()* in
> *string()*, nothing is written and no files are created. With
> *file:write()*, text is always written and files always get created.
>
> Thanks!
> Tim
>
>
> --
> Tim A. Thompson
> Metadata Librarian
> Yale University Library
>
>

[basex-talk] Jobs and file modules

[Tim Thompson]

Hello,

I was experimenting with the jobs module and wondering why there's a
difference between *file:write() *and *file:write-text()* in the query
below:

for $i in 1 to 5
return (
  jobs:eval('
declare variable $iter external;
file:write-text("~/Desktop/job"|| $iter ||".txt", (prof:sleep(5000),
string(current-dateTime(
', map {"iter": $i}
  )
)

With *file:write-text()*, if I don't wrap *current-dateTime()* in *string()*,
nothing is written and no files are created. With *file:write()*, text is
always written and files always get created.

Thanks!
Tim


--
Tim A. Thompson
Metadata Librarian
Yale University Library

Re: [basex-talk] Write output using proc:execute

[Tim Thompson]

Ah, of course. Thank you, Liam! That's what I needed.

All best,
Tim


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Thu, Apr 9, 2020 at 4:31 PM Liam R. E. Quin 
wrote:

> On Thu, 2020-04-09 at 16:00 -0400, Tim Thompson wrote:
> >
> > proc:execute("echo", ("hello!", "> hello.txt"))
>
> You could run, bash -c 'echo hello > hello.txt'
> instead, maybe?
>
>
> This is assuming you are using Linux or the Linux subsystem on Windows,
> or cygwin, or OS X... so bash is available.
>
> --
> Liam Quin, https://www.delightfulcomputing.com/
> Available for XML/Document/Information Architecture/XSLT/
> XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
> Barefoot Web-slave, antique illustrations:  http://www.fromoldbooks.org
>
>

[basex-talk] Write output using proc:execute

[Tim Thompson]

Hello,

Is it possible to redirect output from a system command within the
proc:execute function?

I would like to do something like:

proc:execute("echo", ("hello!", "> hello.txt"))

but the ">" does not seem to be recognized.

Thanks in advance,
Tim


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

Re: [basex-talk] Processing instruction constructors: problem with output

[Tim Thompson]

Thanks for testing, Bridger! You should see the "3" if XSLTForms is
working. I got this to work correctly by changing the BaseX output
serialization method to "xml" instead of "xhtml". I'm not sure why "xhtml"
doesn't work, but XSLTForms converts the document to HTML in the browser
anyway, so this should be okay.

Best,
Tim

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

On Thu, Aug 22, 2019 at 11:35 AM Bridger Dyson-Smith 
wrote:

> Hi Tim -
> A quick test and this is working correctly(?) for me;
> 1. I cloned Alain's repo to `.../webapp/static/`
> 2. I copy-pasta'd your example into `.../webapp/test.xqm`
> 3. After starting the BaseX HTTP server, I hit `localhost:8984` and
> `localhost:8984/test` and had good responses from both; i.e. at `/test`,
> I'm seeing a "Test", a "Placeholder", and a fun little gray-ish ball (but
> no "3" in either browser).
>
> This was with Firefox 60.8.0esr and Chromium 76.0.3809.87.
>
> Is there anything specific I can try to tell you? I'm not very familiar
> with XSLTforms.
> Best,
> Bridger
>
> On Thu, Aug 22, 2019 at 11:16 AM Christian Grün 
> wrote:
>
>> I can’t promise if I can further dive into this. Maybe Alain has some
>> hints?
>>
>>
>> On Thu, Aug 22, 2019 at 5:09 PM Tim Thompson  wrote:
>> >
>> > Okay, so the response panel shows the correct output, but for some
>> reason the stylesheet is not being called. If it's not too much trouble,
>> would you be able to try this with XSLTForms (
>> https://github.com/AlainCouthures/xphoneforms.git) placed in the BaseX
>> webapp static dir?
>> >
>> > Here is an expanded demo query:
>> >
>> >
>> > xquery version "3.1";
>> >
>> > module namespace test = "test";
>> >
>> > declare
>> >   %rest:path("/test")
>> >   %rest:GET
>> >   %output:method("xhtml")
>> > function test:home(
>> >
>> > ) as document-node() {
>> >
>> >   document {
>> > processing-instruction { "xml-stylesheet" } {
>> > 'type="text/xsl"
>> href="/static/xphoneforms/build/xsl/xsltforms.xsl"'
>> > },
>> > processing-instruction { "xsltforms-options" } { 'debug="no"' },
>> > processing-instruction { "css-conversion" } { 'no' }
>> > ,
>> > http://www.w3.org/1999/xhtml; xmlns:xf="
>> http://www.w3.org/2002/xforms;
>> >   xmlns:ev="http://www.w3.org/2001/xml-events; xmlns:my="test"
>> >   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>> xmlns:xsd="http://www.w3.org/2001/XMLSchema;>
>> >
>> >   
>> > TALLER BIBFRAME
>> > http://www.w3.org/2002/xforms; id="main">
>> >   
>> > 
>> >   Test
>> > 
>> >   
>> > 
>> >
>> >
>> >   
>> >   
>> > Placeholder
>> > 
>> > 
>> >   
>> > 
>> >   
>> > 
>> >   }
>> >
>> > };
>> >
>> > --
>> > Tim A. Thompson
>> > Discovery Metadata Librarian
>> > Yale University Library
>> >
>> > On Thu, Aug 22, 2019 at 10:57 AM Christian Grün <
>> christian.gr...@gmail.com> wrote:
>> >>
>> >> Hm… You could check the Response panel in the Network View of your
>> >> browser. The latest version of Chrome, Firefox, Opera, Edge and IE
>> >> should give you the correct result (those are the ones that I have
>> >> tried).
>> >>
>> >>
>> >> > I'm getting the comments in the browser (web console). When I
>> dereference the endpoint with curl, the processing instructions are correct.
>> >> >
>> >> > In the browser, the XForms instructions are not being processed. I
>> don't think XSLTForms would be the culprit, because the stylesheet is not
>> being called.
>> >> >
>> >> > --
>> >> > Tim A. Thompson
>> >> > Discovery Metadata Librarian
>> >> > Yale University Library
>> >> >
>> >> > On Thu, Aug 22, 2019 at 10:37 AM Christian Grün <
>> christian.gr...@gmail.com> wrote:
>> >> >>
>> >> &

Re: [basex-talk] Processing instruction constructors: problem with output

[Tim Thompson]

Thanks for your quick response, Christian. I'm running BaseX 9.2.4.

I'm getting the comments in the browser (web console). When I dereference
the endpoint with curl, the processing instructions are correct.

In the browser, the XForms instructions are not being processed. I don't
think XSLTForms would be the culprit, because the stylesheet is not being
called.

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

On Thu, Aug 22, 2019 at 10:37 AM Christian Grün 
wrote:

> Hi Tim,
>
> I have just tested your endpoint code (with the latest version of
> BaseX), and I receive processing instructions as expected:
>
>  href="static/xphoneforms/build/xsl/xsltforms.xsl"?>
> ...
>
> Which version of BaseX are you using? Are you also getting comments in
> your response when calling the function from the browser or via other
> tools (such as curl), and could it have to do with the way how
> XSLTForms processes the result?
>
> Best,
> Christian
>
>
> On Thu, Aug 22, 2019 at 4:31 PM Tim Thompson  wrote:
> >
> > Hello,
> >
> > I'm trying to output an XForms (XSLTForms) document from a RESTXQ
> endpoint, but the necessary processing instructions are being serialized in
> XML comments in the output, e.g.:
> >
> > 
> >
> >  Am I missing something?
> >
> > XQuery:
> >
> > xquery version "3.1";
> >
> > module namespace test = "test";
> >
> > declare
> >   %rest:path("/test")
> >   %rest:GET
> >   %output:method("xhtml")
> > function test:home() as document-node() {
> >
> >   document {
> > processing-instruction { "xml-stylesheet" } {
> > 'type="text/xsl"
> href="static/xphoneforms/build/xsl/xsltforms.xsl"'
> > },
> > processing-instruction { "xsltforms-options" } { 'debug="no"' },
> > processing-instruction { "css-conversion" } { 'no' }
> > ,
> > http://www.w3.org/1999/xhtml; xmlns:xforms="
> http://www.w3.org/2002/xforms;
> >   xmlns:ev="http://www.w3.org/2001/xml-events; xmlns:my="test"
> >   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xmlns:xsd="http://www.w3.org/2001/XMLSchema;>
> >
> >   
> > Test
> >   
> >   
> > Placeholder
> >   
> > 
> >   }
> > };
> >
> >
> > --
> > Tim A. Thompson
> > Discovery Metadata Librarian
> > Yale University Library
> >
>

[basex-talk] Processing instruction constructors: problem with output

[Tim Thompson]

Hello,

I'm trying to output an XForms (XSLTForms) document from a RESTXQ endpoint,
but the necessary processing instructions are being serialized in XML
comments in the output, e.g.:



 Am I missing something?

XQuery:

xquery version "3.1";

module namespace test = "test";

declare
  %rest:path("/test")
  %rest:GET
  %output:method("xhtml")
function test:home() as document-node() {

  document {
processing-instruction { "xml-stylesheet" } {
'type="text/xsl" href="static/xphoneforms/build/xsl/xsltforms.xsl"'
},
processing-instruction { "xsltforms-options" } { 'debug="no"' },
processing-instruction { "css-conversion" } { 'no' }
,
http://www.w3.org/1999/xhtml; xmlns:xforms="
http://www.w3.org/2002/xforms;
  xmlns:ev="http://www.w3.org/2001/xml-events; xmlns:my="test"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xmlns:xsd="
http://www.w3.org/2001/XMLSchema;>

  
Test
  
  
Placeholder
  

  }
};


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

Re: [basex-talk] Forcing download of JSON doc

[Tim Thompson]

Thanks for the tip! By declaring the %output:method("json") annotation AND
calling the serialize function, I was basically reserializing the data.

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

On Thu, Aug 1, 2019 at 8:55 AM Marco Lettere  wrote:

> Maybe using json:serialize() rather than serialize?
>
> In general if output:method is set to Json it should be sufficient to
> return map{} or array{} or alternatively the XML version of a json document
> without having to explicitly serialize ...
>
> M.
>
> On 01/08/19 14:36, Tim Thompson wrote:
>
> Thank you, Marco. So, the download issue seems to be with XForms
> (XSLTForms) rather than BaseX. If I do a file upload from a plain HTML
> form, the file download is triggered correctly. However, even though I have
> set "application/json" as the Content-Type, the contents of the downloaded
> file are an escaped string rather than a serialized JSON object: e.g.,
>
> "{\n  \"key\": \"value\"\n}"
>
> Is there a way to produce a properly serialized JSON file for download?
>
> --
> Tim A. Thompson
> Discovery Metadata Librarian
> Yale University Library
>
> On Thu, Aug 1, 2019 at 3:03 AM Marco Lettere  wrote:
>
>> Hi Tim,
>> downloading from a browser when posting a form is usually a bit tricky.
>> For me it always ended up with the creation of an artificial 
>> element with an href crafted from the form parameters and with an attribute
>> download="filename.ext" added to it.
>> Anyway you could try with returning a "forced" application/octet-stream
>> Content-Type header 
>>
>> M.
>>
>> On 01/08/19 05:00, Tim Thompson wrote:
>>
>> Hello,
>>
>> I am posting a JSON doc to a RESTXQ function from XForms and trying to
>> set a Content-Disposition header to trigger a file download. Is this
>> possible to do without first saving the JSON doc as a file? Right now, the
>> RESTXQ function returns the correct headers, but does not force a file
>> download dialog.
>>
>> Thanks in advance!
>> Tim
>>
>> --
>> Query:
>>
>> xquery version "3.1";
>>
>> module namespace wrap = "http://example.org/wrap;;
>>
>> declare
>>   %rest:path("/wrap")
>>   %rest:POST("{$data}")
>>   %output:method("json")
>> function wrap:process(
>>   $data as node()
>> ) as item()* {
>>   let $name := data($data/data/upload/file/@name)
>>   let $json := json-to-xml($data/data/upload/file)
>>   let $header :=
>> ``[attachment; filename="`{$name}`";]``
>>   return
>> (
>> 
>>   > status="200" message="OK">
>> >   name="Content-Type"
>>   value="application/json"/>
>> >   name="Content-Disposition"
>>   value="{$header}"/>
>>   
>> ,
>>
>> http://www.w3.org/2005/xpath-functions;>
>> >   key="Profile">
>>   > key="resourceTemplates">{$json}
>>   > key="title">Wrapper_Profile
>>   > key="id">Wrapper:Profile:BF2
>>   > key="description">Profile wrapper for resource
>> templates
>>   > key="author">Wrapper
>>   > key="schema">
>> https://ld4p.github.io/sinopia/schemas/0.0.9/profile.json
>>   > key="date">{current-dateTime()}
>> 
>>  => xml-to-json() => parse-json() => serialize(map {'method':
>> 'json', 'use-character-maps' : map {'/' : '/'}})
>> )
>> };
>>
>>
>> --
>> Tim A. Thompson
>> Discovery Metadata Librarian
>> Yale University Library
>>
>>
>>
>

Re: [basex-talk] Forcing download of JSON doc

[Tim Thompson]

Thank you, Marco. So, the download issue seems to be with XForms
(XSLTForms) rather than BaseX. If I do a file upload from a plain HTML
form, the file download is triggered correctly. However, even though I have
set "application/json" as the Content-Type, the contents of the downloaded
file are an escaped string rather than a serialized JSON object: e.g.,

"{\n  \"key\": \"value\"\n}"

Is there a way to produce a properly serialized JSON file for download?

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

On Thu, Aug 1, 2019 at 3:03 AM Marco Lettere  wrote:

> Hi Tim,
> downloading from a browser when posting a form is usually a bit tricky.
> For me it always ended up with the creation of an artificial 
> element with an href crafted from the form parameters and with an attribute
> download="filename.ext" added to it.
> Anyway you could try with returning a "forced" application/octet-stream
> Content-Type header 
>
> M.
>
> On 01/08/19 05:00, Tim Thompson wrote:
>
> Hello,
>
> I am posting a JSON doc to a RESTXQ function from XForms and trying to set
> a Content-Disposition header to trigger a file download. Is this possible
> to do without first saving the JSON doc as a file? Right now, the RESTXQ
> function returns the correct headers, but does not force a file download
> dialog.
>
> Thanks in advance!
> Tim
>
> --
> Query:
>
> xquery version "3.1";
>
> module namespace wrap = "http://example.org/wrap;;
>
> declare
>   %rest:path("/wrap")
>   %rest:POST("{$data}")
>   %output:method("json")
> function wrap:process(
>   $data as node()
> ) as item()* {
>   let $name := data($data/data/upload/file/@name)
>   let $json := json-to-xml($data/data/upload/file)
>   let $header :=
> ``[attachment; filename="`{$name}`";]``
>   return
> (
> 
>status="200" message="OK">
>name="Content-Type"
>   value="application/json"/>
>name="Content-Disposition"
>   value="{$header}"/>
>   
> ,
>
> http://www.w3.org/2005/xpath-functions;>
>key="Profile">
>key="resourceTemplates">{$json}
>key="title">Wrapper_Profile
>key="id">Wrapper:Profile:BF2
>key="description">Profile wrapper for resource
> templates
>key="author">Wrapper
>key="schema">
> https://ld4p.github.io/sinopia/schemas/0.0.9/profile.json
>key="date">{current-dateTime()}
> 
>  => xml-to-json() => parse-json() => serialize(map {'method':
> 'json', 'use-character-maps' : map {'/' : '/'}})
> )
> };
>
>
> --
> Tim A. Thompson
> Discovery Metadata Librarian
> Yale University Library
>
>
>

[basex-talk] Forcing download of JSON doc

[Tim Thompson]

Hello,

I am posting a JSON doc to a RESTXQ function from XForms and trying to set
a Content-Disposition header to trigger a file download. Is this possible
to do without first saving the JSON doc as a file? Right now, the RESTXQ
function returns the correct headers, but does not force a file download
dialog.

Thanks in advance!
Tim

--
Query:

xquery version "3.1";

module namespace wrap = "http://example.org/wrap;;

declare
  %rest:path("/wrap")
  %rest:POST("{$data}")
  %output:method("json")
function wrap:process(
  $data as node()
) as item()* {
  let $name := data($data/data/upload/file/@name)
  let $json := json-to-xml($data/data/upload/file)
  let $header :=
``[attachment; filename="`{$name}`";]``
  return
(

  


  
,

http://www.w3.org/2005/xpath-functions;>

  {$json}
  Wrapper_Profile
  Wrapper:Profile:BF2
  Profile wrapper for resource
templates
  Wrapper
  
https://ld4p.github.io/sinopia/schemas/0.0.9/profile.json
  {current-dateTime()}

 => xml-to-json() => parse-json() => serialize(map {'method':
'json', 'use-character-maps' : map {'/' : '/'}})
)
};


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

[basex-talk] Signature of db:add

[Tim Thompson]

Hello,

The two argument signature of the db:add function does not seem to work (
http://docs.basex.org/wiki/Database_Module#db:add). Is this still valid in
BaseX?

When I try something like db:add("test", ), I get an error:

[FODC0007] Resource path '' is invalid.

Thanks!

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

Re: [basex-talk] BaseXdb/basex] Packaging: Java and XQuery code (#1413)

[Tim Thompson]

Thanks, I did try stream:materialize, but that didn't seem to help.
Regarding the "limitation" of the new combined approach, I think that may
actually be a plus. In my case, it forced me to make my code a bit more
modular.

Tim

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

On Wed, Oct 25, 2017 at 11:25 AM, Christian Grün 
wrote:

> Hi Tim,
>
> Thanks for the hint, and the intent to create a combined package for
> your code (our approach seems to be too limited indeed).
>
> Once again my day was shorter than I expected, so I’ll do my best to
> check out your project tomorrow.
>
> > When I try to use the BaseX fetch:text function
> > to retrieve an RDF text file and pass it to my combined module parser
> > function, I get an error (in the the latest BaseX snapshot: beta
> a48337f):
> > Function parseTrigDoc(xs:string) cannot be called with (xs:string).
>
> My spontaneous guess is that fetch:text returns a “streamable” string
> (its contents will only be generated if they are requested), and our
> Java conversion mapping seem not be prepared to handle these types of
> strings. Actually they should be, so I’ll check how this can be fixed.
> In the meanwhile, you should be able to work around the restriction by
> explicitly requesting the string via
> stream:materialize(fetch:text(...)).
>
> Hope this helps,
> Christian
>
> [1] http://docs.basex.org/wiki/Fetch_Module#fetch:text
> [2] http://docs.basex.org/wiki/Streaming_Module
>

Re: [basex-talk] BaseXdb/basex] Packaging: Java and XQuery code (#1413)

[Tim Thompson]

One more related question. When I try to use the BaseX fetch:text function
to retrieve an RDF text file and pass it to my combined module parser
function, I get an error (in the the latest BaseX snapshot: beta
a48337f): Function
parseTrigDoc(xs:string) cannot be called with (xs:string). However, if I
change from fetch:text to unparsed-text, I am able to retrieve the file and
parse it successfully. Is it possible to use fetch:text here[1]?

Thanks again.

Tim

[1] https://github.com/metadataframes/basex-rdf/blob/master/src/triples.xq


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Tue, Oct 24, 2017 at 5:25 PM, Tim Thompson <timat...@gmail.com> wrote:

> Hi, Christian,
>
> I had a chance to try out the combined module feature this afternoon.
> Copies of the combined JAR file and related files are here:
> https://github.com/timathom/basex-rdf.
>
> I was able to create a basic module, but I did run into a few hiccups
> along the way:
>
>1. From my XQuery module, I want to call a pair of XSLT stylesheets,
>so I tried adding these to the JAR as well. However, when I import the
>combined JAR file, the XSLT files are not copied over. If I copy them
>manually into the BaseXRepo folder, then it works.
>2. In my previous experiments with the EXPath repo approach, I was
>able to import the Java module while also defining a separate namespace for
>the XQuery wrapper module itself. Thus, I was able to define additional
>XQuery functions in the namespace of the wrapper module. With the new
>combined approach, however, it seems that each XQuery function must
>correspond to a Java function, and the namespaces must match (e.g., "
>http://basex.org/modules/rdf/Triples
><http://basex.org/modules/rdf/Triples>" and "java:
>org.basex.modules.rdf.Triples"). I had to remove my additional XQuery
>functions from the wrapper because BaseX was throwing an error that the
>functions were not Java functions.
>
> Just let me know if I can provide any other feedback that would be helpful.
>
> Best,
> Tim
>
>
> --
> Tim A. Thompson
> Discovery Metadata Librarian
> Yale University Library
>
>
> On Tue, Oct 24, 2017 at 12:36 PM, Christian Grün <
> christian.gr...@gmail.com> wrote:
>
>> Hi Tim,
>>
>> I’ll have a look soon, probably tomorrow. Out of interest: Have you
>> been successful using our combined approach?
>>
>> Best,
>> Christian
>>
>>
>>
>> On Tue, Oct 24, 2017 at 4:16 PM, Tim Thompson <timat...@gmail.com> wrote:
>> > Thanks, Christian. I had a quick question about the expath-pkg.xml file
>> > (attached). I have two XSLT files that I want to list separately, along
>> with
>> > listing Saxon-HE (version 9.8.x) as a dependency. However, when I add a
>> > second dependency element and try to import the package, I get an error:
>> >
>> > [bxerr:BXRE0009] Package version is not supported.
>> >
>> > Looking at the EXPath package schema[1], it seems as though this should
>> be
>> > valid (the dependency element should be repeatable).
>> >
>> > Tim
>> >
>> > [1] http://expath.org/spec/pkg#structure
>> >
>> > --
>> > Tim A. Thompson
>> > Discovery Metadata Librarian
>> > Yale University Library
>> >
>> >
>> > On Fri, Oct 20, 2017 at 2:25 PM, Christian Grün <
>> christian.gr...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Tim,
>> >>
>> >> Thanks for the link to your RDF library.
>> >>
>> >> I’m glad to hear that the new packaging mechanism has already found
>> >> watchers. I have just uploaded a new snapshot, and I would be glad to
>> >> have your feedback as first tester ;)
>> >>
>> >> > but I wanted to be sure I was going about it correctly because it
>> still
>> >> > seems a little like magic to me. I am also wondering whether this new
>> >> > feature is something I might want to take advantage of.
>> >>
>> >> I haven’t run any queries yet with your module, but the package
>> >> structure looks fine.
>> >>
>> >> Our experience over the years is that the EXPath packaging mechanism
>> >> is a bit difficult to understand, and not as portable as we believe it
>> >> could be. It would be nice, for example, if a package could be
>> >> distributed without implementation-specific details inside. But it’s
>> >> always easy to complain, and it takes a lot of 

Re: [basex-talk] BaseXdb/basex] Packaging: Java and XQuery code (#1413)

[Tim Thompson]

Hi, Christian,

I had a chance to try out the combined module feature this afternoon.
Copies of the combined JAR file and related files are here:
https://github.com/timathom/basex-rdf.

I was able to create a basic module, but I did run into a few hiccups along
the way:

   1. From my XQuery module, I want to call a pair of XSLT stylesheets, so
   I tried adding these to the JAR as well. However, when I import the
   combined JAR file, the XSLT files are not copied over. If I copy them
   manually into the BaseXRepo folder, then it works.
   2. In my previous experiments with the EXPath repo approach, I was able
   to import the Java module while also defining a separate namespace for the
   XQuery wrapper module itself. Thus, I was able to define additional XQuery
   functions in the namespace of the wrapper module. With the new combined
   approach, however, it seems that each XQuery function must correspond to a
   Java function, and the namespaces must match (e.g., "
   http://basex.org/modules/rdf/Triples; and
   "java:org.basex.modules.rdf.Triples"). I had to remove my additional XQuery
   functions from the wrapper because BaseX was throwing an error that the
   functions were not Java functions.

Just let me know if I can provide any other feedback that would be helpful.

Best,
Tim


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Tue, Oct 24, 2017 at 12:36 PM, Christian Grün <christian.gr...@gmail.com>
wrote:

> Hi Tim,
>
> I’ll have a look soon, probably tomorrow. Out of interest: Have you
> been successful using our combined approach?
>
> Best,
> Christian
>
>
>
> On Tue, Oct 24, 2017 at 4:16 PM, Tim Thompson <timat...@gmail.com> wrote:
> > Thanks, Christian. I had a quick question about the expath-pkg.xml file
> > (attached). I have two XSLT files that I want to list separately, along
> with
> > listing Saxon-HE (version 9.8.x) as a dependency. However, when I add a
> > second dependency element and try to import the package, I get an error:
> >
> > [bxerr:BXRE0009] Package version is not supported.
> >
> > Looking at the EXPath package schema[1], it seems as though this should
> be
> > valid (the dependency element should be repeatable).
> >
> > Tim
> >
> > [1] http://expath.org/spec/pkg#structure
> >
> > --
> > Tim A. Thompson
> > Discovery Metadata Librarian
> > Yale University Library
> >
> >
> > On Fri, Oct 20, 2017 at 2:25 PM, Christian Grün <
> christian.gr...@gmail.com>
> > wrote:
> >>
> >> Hi Tim,
> >>
> >> Thanks for the link to your RDF library.
> >>
> >> I’m glad to hear that the new packaging mechanism has already found
> >> watchers. I have just uploaded a new snapshot, and I would be glad to
> >> have your feedback as first tester ;)
> >>
> >> > but I wanted to be sure I was going about it correctly because it
> still
> >> > seems a little like magic to me. I am also wondering whether this new
> >> > feature is something I might want to take advantage of.
> >>
> >> I haven’t run any queries yet with your module, but the package
> >> structure looks fine.
> >>
> >> Our experience over the years is that the EXPath packaging mechanism
> >> is a bit difficult to understand, and not as portable as we believe it
> >> could be. It would be nice, for example, if a package could be
> >> distributed without implementation-specific details inside. But it’s
> >> always easy to complain, and it takes a lot of time to get things
> >> going.
> >>
> >> For our own use cases, the simple packaging mechanism is just good
> >> enough. It would be nice if we could add some versioning support in
> >> the future, though.
> >>
> >> Cheers
> >> Christian
> >>
> >>
> >>
> >> > The module (https://github.com/timathom/basex-rdf) uses Günther
> >> > Rademacher's
> >> > REx Parser Generator[1] to create a Java parser for the RDF
> TriG/Turtle
> >> > syntax[2], using the BaseX extension function option.
> >> >
> >> > Following the BaseX documentation[3], I have created a XAR package
> that
> >> > includes the JAR file and an XQuery wrapper. The wrapper calls the
> >> > function
> >> > exposed by the Java module (import module namespace trig =
> >> > "org.basex.modules.rdf.TriG";) and then processes the raw parse tree,
> >> > overlaying some additional abstractions (I confess that I cheated and
> >> > used
> >> > XSLT for most of the processing functionality).
> >> >
> >> > Because of the built-in BaseX extension function, this approach seems
> >> > different from a straightforward "combined" module approach, but,
> again,
> >> > I
> >> > wanted to ask whether the way I'm doing this seems correct.
> >> >
> >> > Thanks in advance.
> >> >
> >> > Tim
> >> >
> >> > [1] http://www.bottlecaps.de/rex/
> >> > [2] https://www.w3.org/TR/trig/
> >> > [3] docs.basex.org/wiki/Repository
> >> >
> >> > --
> >> > Tim A. Thompson
> >> > Discovery Metadata Librarian
> >> > Yale University Library
> >> >
> >
> >
>

Re: [basex-talk] BaseXdb/basex] Packaging: Java and XQuery code (#1413)

[Tim Thompson]

Thanks, Christian. I had a quick question about the expath-pkg.xml file
(attached). I have two XSLT files that I want to list separately, along
with listing Saxon-HE (version 9.8.x) as a dependency. However, when I add
a second dependency element and try to import the package, I get an error:

[bxerr:BXRE0009] Package version is not supported.

Looking at the EXPath package schema[1], it seems as though this should be
valid (the dependency element should be repeatable).

Tim

[1] http://expath.org/spec/pkg#structure

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Fri, Oct 20, 2017 at 2:25 PM, Christian Grün 
wrote:

> Hi Tim,
>
> Thanks for the link to your RDF library.
>
> I’m glad to hear that the new packaging mechanism has already found
> watchers. I have just uploaded a new snapshot, and I would be glad to
> have your feedback as first tester ;)
>
> > but I wanted to be sure I was going about it correctly because it still
> > seems a little like magic to me. I am also wondering whether this new
> > feature is something I might want to take advantage of.
>
> I haven’t run any queries yet with your module, but the package
> structure looks fine.
>
> Our experience over the years is that the EXPath packaging mechanism
> is a bit difficult to understand, and not as portable as we believe it
> could be. It would be nice, for example, if a package could be
> distributed without implementation-specific details inside. But it’s
> always easy to complain, and it takes a lot of time to get things
> going.
>
> For our own use cases, the simple packaging mechanism is just good
> enough. It would be nice if we could add some versioning support in
> the future, though.
>
> Cheers
> Christian
>
>
>
> > The module (https://github.com/timathom/basex-rdf) uses Günther
> Rademacher's
> > REx Parser Generator[1] to create a Java parser for the RDF TriG/Turtle
> > syntax[2], using the BaseX extension function option.
> >
> > Following the BaseX documentation[3], I have created a XAR package that
> > includes the JAR file and an XQuery wrapper. The wrapper calls the
> function
> > exposed by the Java module (import module namespace trig =
> > "org.basex.modules.rdf.TriG";) and then processes the raw parse tree,
> > overlaying some additional abstractions (I confess that I cheated and
> used
> > XSLT for most of the processing functionality).
> >
> > Because of the built-in BaseX extension function, this approach seems
> > different from a straightforward "combined" module approach, but, again,
> I
> > wanted to ask whether the way I'm doing this seems correct.
> >
> > Thanks in advance.
> >
> > Tim
> >
> > [1] http://www.bottlecaps.de/rex/
> > [2] https://www.w3.org/TR/trig/
> > [3] docs.basex.org/wiki/Repository
> >
> > --
> > Tim A. Thompson
> > Discovery Metadata Librarian
> > Yale University Library
> >
>

http://expath.org/ns/pkg;
name="X-Triples"
version="0.1"
abbrev="jar"
spec="1.0">
REx Parser Generator RDF parser for Turtle, N-Triples, TriG



https://metadatafram.es/basex/modules/rdf/triples/
wrapper.xq


https://metadatafram.es/basex/modules/rdf/triples/
process.xsl
postprocess.xsl

Re: [basex-talk] BaseXdb/basex] Packaging: Java and XQuery code (#1413)

[Tim Thompson]

Christian, all,

The new "combined" packaging option (
https://github.com/BaseXdb/basex/issues/1413) seems like an interesting
enhancement.

I have been working on my first BaseX repository module/package (though
unfortunately I am not yet fluent in Java). It is working perfectly so far,
but I wanted to be sure I was going about it correctly because it still
seems a little like magic to me. I am also wondering whether this new
feature is something I might want to take advantage of.

The module (https://github.com/timathom/basex-rdf) uses Günther
Rademacher's REx Parser Generator[1] to create a Java parser for the RDF
TriG/Turtle syntax[2], using the BaseX extension function option.

Following the BaseX documentation[3], I have created a XAR package that
includes the JAR file and an XQuery wrapper. The wrapper calls the function
exposed by the Java module (import module namespace trig =
"org.basex.modules.rdf.TriG";) and then processes the raw parse tree,
overlaying some additional abstractions (I confess that I cheated and used
XSLT for most of the processing functionality).

Because of the built-in BaseX extension function, this approach seems
different from a straightforward "combined" module approach, but, again, I
wanted to ask whether the way I'm doing this seems correct.

Thanks in advance.

Tim

[1] http://www.bottlecaps.de/rex/
[2] https://www.w3.org/TR/trig/
[3] docs.basex.org/wiki/Repository

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

[basex-talk] BaseX continuous integration

[Tim Thompson]

Hello,

Does anyone have an example they could share of an app using the BaseX Unit
Module deployed on a continuous integration server (preferably Travis CI)?

Thanks in advance.

Tim

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

Re: [basex-talk] RESTXQ custom response

[Tim Thompson]

Christian,

Thinking through this more, I don't believe anything needs to change in the
current BaseX authentication handling. For my use case, I can just use the
default BaseX Digest authentication, or else use a custom HTTP header.

I apologize for the trouble!

Thanks again,
Tim

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Sun, Aug 20, 2017 at 1:10 PM, Christian Grün <christian.gr...@gmail.com>
wrote:

> I must confess I’ll need some time to get through this (I haven’t
> spent too much time with the Custom option so far). If everything
> works out, I’ll be able to do so in the upcoming week. If you manage
> to provide me with an SSCCE that show exactly what would need to
> happen, that would be great!
>
> Thanks,
> Christian
>
>
> On Sat, Aug 19, 2017 at 4:54 AM, Tim Thompson <timat...@gmail.com> wrote:
> > Actually, I was wrong. The issue seems to be with the 401 Unauthorized
> > status needed with Digest authentication. I can control the header value
> > using both methods when the status is 200 OK, but BaseX does not allow
> me to
> > modify the WWW-Authenticate header when the status is 401. Is this
> something
> > that could be changed?
> >
> >
> > --
> > Tim A. Thompson
> > Discovery Metadata Librarian
> > Yale University Library
> >
> >
> > On Fri, Aug 18, 2017 at 10:26 PM, Tim Thompson <timat...@gmail.com>
> wrote:
> >>
> >> Christian,
> >>
> >> Here is a minimal example to try to illustrate what I am referring to:
> >>
> >> https://bibfram.es/basex/static/header-test.xhtml
> >>
> >> The first case (using the web:response-header() function) produces the
> >> desired response. You should see something like this after clicking
> "Test"
> >> (at least the first time, before caching):
> >>
> >>  WWW-Authenticate Digest realm="BaseX",
> >> nonce="35B2F3011682300F36AD37048B7B8560"  
> >>
> >> The second case (using custom elements) does not produce the desired
> >> response. You should see something like this after clicking "Test":
> >>
> >>  WWW-Authenticate Custom realm="BaseX"
> >>  
> >>
> >> So the two methods are not producing the same output.
> >>
> >>
> >> --
> >> Tim A. Thompson
> >> Discovery Metadata Librarian
> >> Yale University Library
> >>
> >> On Fri, Aug 18, 2017 at 7:28 PM, Christian Grün
> >> <christian.gr...@gmail.com> wrote:
> >>>
> >>> Hm, I still need to understand: Why can't you set WWW-Authenticate via
> >>> standard response elements if you can do it with web:response-header
> (which
> >>> simply generates elements that you could write by yourself)?
> >>>
> >>>
> >>>
> >>> Am 19.08.2017 1:04 vorm. schrieb "Tim Thompson" <timat...@gmail.com>:
> >>>
> >>> Yes, but that brings me back to my original issue. The standard RESTXQ
> >>> response headers let me set the HTTP status, but they don't let me
> override
> >>> some default header values: namely, WWW-Authenticate when BaseX is
> >>> configured for "Custom" authentication. The web:response-header
> function
> >>> does let me modify the header, but it does not let me set the HTTP
> status.
> >>>
> >>> So, there are two different solutions here; each solves part of my
> >>> problem, but neither one solves the whole problem :(
> >>>
> >>> Tim
> >>>
> >>> --
> >>> Tim A. Thompson
> >>> Discovery Metadata Librarian
> >>> Yale University Library
> >>>
> >>>
> >>> On Fri, Aug 18, 2017 at 4:46 PM, Christian Grün
> >>> <christian.gr...@gmail.com> wrote:
> >>>>
> >>>> True. In many cases, you are probably more flexible by using the
> >>>> standard RESTXQ response headers.
> >>>>
> >>>> On Fri, Aug 18, 2017 at 4:37 PM, Tim Thompson <timat...@gmail.com>
> >>>> wrote:
> >>>> > Right, the status can be set when using the http:response element
> >>>> > directly,
> >>>> > but the web:response-header function does not seem to provide access
> >>>> > to
> >>>> > that. It only allows one to set new headers, not set the status of
> the
> >>>> > response.
> >>>

Re: [basex-talk] RESTXQ custom response

[Tim Thompson]

Actually, I was wrong. The issue seems to be with the 401 Unauthorized
status needed with Digest authentication. I can control the header value
using both methods when the status is 200 OK, but BaseX does not allow me
to modify the WWW-Authenticate header when the status is 401. Is this
something that could be changed?


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Fri, Aug 18, 2017 at 10:26 PM, Tim Thompson <timat...@gmail.com> wrote:

> Christian,
>
> Here is a minimal example to try to illustrate what I am referring to:
>
> https://bibfram.es/basex/static/header-test.xhtml
>
> The first case (using the web:response-header() function) produces the
> desired response. You should see something like this after clicking "Test"
> (at least the first time, before caching):
>
>  WWW-Authenticate Digest realm="BaseX", nonce="
> 35B2F3011682300F36AD37048B7B8560"  
>
> The second case (using custom elements) does not produce the desired
> response. You should see something like this after clicking "Test":
>
>  WWW-Authenticate Custom realm="BaseX"
>  
>
> So the two methods are not producing the same output.
>
>
> --
> Tim A. Thompson
> Discovery Metadata Librarian
> Yale University Library
>
> On Fri, Aug 18, 2017 at 7:28 PM, Christian Grün <christian.gr...@gmail.com
> > wrote:
>
>> Hm, I still need to understand: Why can't you set WWW-Authenticate via
>> standard response elements if you can do it with web:response-header (which
>> simply generates elements that you could write by yourself)?
>>
>>
>>
>> Am 19.08.2017 1:04 vorm. schrieb "Tim Thompson" <timat...@gmail.com>:
>>
>> Yes, but that brings me back to my original issue. The standard RESTXQ
>> response headers let me set the HTTP status, but they don't let me override
>> some default header values: namely, WWW-Authenticate when BaseX is
>> configured for "Custom" authentication. The web:response-header function
>> does let me modify the header, but it does not let me set the HTTP status.
>>
>> So, there are two different solutions here; each solves part of my
>> problem, but neither one solves the whole problem :(
>>
>> Tim
>>
>> --
>> Tim A. Thompson
>> Discovery Metadata Librarian
>> Yale University Library
>>
>>
>> On Fri, Aug 18, 2017 at 4:46 PM, Christian Grün <
>> christian.gr...@gmail.com> wrote:
>>
>>> True. In many cases, you are probably more flexible by using the
>>> standard RESTXQ response headers.
>>>
>>> On Fri, Aug 18, 2017 at 4:37 PM, Tim Thompson <timat...@gmail.com>
>>> wrote:
>>> > Right, the status can be set when using the http:response element
>>> directly,
>>> > but the web:response-header function does not seem to provide access to
>>> > that. It only allows one to set new headers, not set the status of the
>>> > response.
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Tim A. Thompson
>>> > Discovery Metadata Librarian
>>> > Yale University Library
>>> >
>>> >
>>> > On Fri, Aug 18, 2017 at 9:23 AM, Christian Grün <
>>> christian.gr...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi Tim,
>>> >>
>>> >> That should be possible as well:
>>> >>
>>> >>   
>>> >> 
>>> >>   
>>> >>
>>> >> Cheers,
>>> >> Christian
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Aug 18, 2017 at 3:20 PM, Tim Thompson <timat...@gmail.com>
>>> wrote:
>>> >> > Hi, Christian,
>>> >> >
>>> >> > Yes, it does. The only issue is that there doesn't seem to be a way
>>> to
>>> >> > set
>>> >> > the HTTP status and message via web:response-header, is that right?
>>> This
>>> >> > would be a great feature to have :)
>>> >> >
>>> >> > Tim
>>> >> >
>>> >> > --
>>> >> > Tim A. Thompson
>>> >> > Discovery Metadata Librarian
>>> >> > Yale University Library
>>> >> >
>>> >> >
>>> >> > On Fri, Aug 18, 2017 at 4:25 AM, Christian Grün
>>> >> > <christian.gr...@gmail.com>
>>> >>

Re: [basex-talk] RESTXQ custom response

[Tim Thompson]

Christian,

Here is a minimal example to try to illustrate what I am referring to:

https://bibfram.es/basex/static/header-test.xhtml

The first case (using the web:response-header() function) produces the
desired response. You should see something like this after clicking "Test"
(at least the first time, before caching):

 WWW-Authenticate Digest realm="BaseX",
nonce="35B2F3011682300F36AD37048B7B8560"  

The second case (using custom elements) does not produce the desired
response. You should see something like this after clicking "Test":

 WWW-Authenticate Custom realm="BaseX" 


So the two methods are not producing the same output.


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

On Fri, Aug 18, 2017 at 7:28 PM, Christian Grün <christian.gr...@gmail.com>
wrote:

> Hm, I still need to understand: Why can't you set WWW-Authenticate via
> standard response elements if you can do it with web:response-header (which
> simply generates elements that you could write by yourself)?
>
>
>
> Am 19.08.2017 1:04 vorm. schrieb "Tim Thompson" <timat...@gmail.com>:
>
> Yes, but that brings me back to my original issue. The standard RESTXQ
> response headers let me set the HTTP status, but they don't let me override
> some default header values: namely, WWW-Authenticate when BaseX is
> configured for "Custom" authentication. The web:response-header function
> does let me modify the header, but it does not let me set the HTTP status.
>
> So, there are two different solutions here; each solves part of my
> problem, but neither one solves the whole problem :(
>
> Tim
>
> --
> Tim A. Thompson
> Discovery Metadata Librarian
> Yale University Library
>
>
> On Fri, Aug 18, 2017 at 4:46 PM, Christian Grün <christian.gr...@gmail.com
> > wrote:
>
>> True. In many cases, you are probably more flexible by using the
>> standard RESTXQ response headers.
>>
>> On Fri, Aug 18, 2017 at 4:37 PM, Tim Thompson <timat...@gmail.com> wrote:
>> > Right, the status can be set when using the http:response element
>> directly,
>> > but the web:response-header function does not seem to provide access to
>> > that. It only allows one to set new headers, not set the status of the
>> > response.
>> >
>> >
>> >
>> >
>> > --
>> > Tim A. Thompson
>> > Discovery Metadata Librarian
>> > Yale University Library
>> >
>> >
>> > On Fri, Aug 18, 2017 at 9:23 AM, Christian Grün <
>> christian.gr...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Tim,
>> >>
>> >> That should be possible as well:
>> >>
>> >>   
>> >> 
>> >>   
>> >>
>> >> Cheers,
>> >> Christian
>> >>
>> >>
>> >>
>> >> On Fri, Aug 18, 2017 at 3:20 PM, Tim Thompson <timat...@gmail.com>
>> wrote:
>> >> > Hi, Christian,
>> >> >
>> >> > Yes, it does. The only issue is that there doesn't seem to be a way
>> to
>> >> > set
>> >> > the HTTP status and message via web:response-header, is that right?
>> This
>> >> > would be a great feature to have :)
>> >> >
>> >> > Tim
>> >> >
>> >> > --
>> >> > Tim A. Thompson
>> >> > Discovery Metadata Librarian
>> >> > Yale University Library
>> >> >
>> >> >
>> >> > On Fri, Aug 18, 2017 at 4:25 AM, Christian Grün
>> >> > <christian.gr...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi Tim,
>> >> >>
>> >> >> I am glad to hear the response header output does its job.
>> >> >>
>> >> >> As the function does nothing else than creating a response header
>> >> >> (unfortunately with this standard caching directive – I think I will
>> >> >> remove it along with BaseX 8.7), you could try to replace your
>> >> >> response elements with the function’s result. Does this help?
>> >> >>
>> >> >> Christian
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Thu, Aug 17, 2017 at 9:21 PM, Tim Thompson <timat...@gmail.com>
>> >> >> wrote:
>> >> >> > Using the web:response-header() function from the BaseX Web Modu

Re: [basex-talk] RESTXQ custom response

[Tim Thompson]

Yes, but that brings me back to my original issue. The standard RESTXQ
response headers let me set the HTTP status, but they don't let me override
some default header values: namely, WWW-Authenticate when BaseX is
configured for "Custom" authentication. The web:response-header function
does let me modify the header, but it does not let me set the HTTP status.

So, there are two different solutions here; each solves part of my problem,
but neither one solves the whole problem :(

Tim

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Fri, Aug 18, 2017 at 4:46 PM, Christian Grün <christian.gr...@gmail.com>
wrote:

> True. In many cases, you are probably more flexible by using the
> standard RESTXQ response headers.
>
> On Fri, Aug 18, 2017 at 4:37 PM, Tim Thompson <timat...@gmail.com> wrote:
> > Right, the status can be set when using the http:response element
> directly,
> > but the web:response-header function does not seem to provide access to
> > that. It only allows one to set new headers, not set the status of the
> > response.
> >
> >
> >
> >
> > --
> > Tim A. Thompson
> > Discovery Metadata Librarian
> > Yale University Library
> >
> >
> > On Fri, Aug 18, 2017 at 9:23 AM, Christian Grün <
> christian.gr...@gmail.com>
> > wrote:
> >>
> >> Hi Tim,
> >>
> >> That should be possible as well:
> >>
> >>   
> >> 
> >>   
> >>
> >> Cheers,
> >> Christian
> >>
> >>
> >>
> >> On Fri, Aug 18, 2017 at 3:20 PM, Tim Thompson <timat...@gmail.com>
> wrote:
> >> > Hi, Christian,
> >> >
> >> > Yes, it does. The only issue is that there doesn't seem to be a way to
> >> > set
> >> > the HTTP status and message via web:response-header, is that right?
> This
> >> > would be a great feature to have :)
> >> >
> >> > Tim
> >> >
> >> > --
> >> > Tim A. Thompson
> >> > Discovery Metadata Librarian
> >> > Yale University Library
> >> >
> >> >
> >> > On Fri, Aug 18, 2017 at 4:25 AM, Christian Grün
> >> > <christian.gr...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Tim,
> >> >>
> >> >> I am glad to hear the response header output does its job.
> >> >>
> >> >> As the function does nothing else than creating a response header
> >> >> (unfortunately with this standard caching directive – I think I will
> >> >> remove it along with BaseX 8.7), you could try to replace your
> >> >> response elements with the function’s result. Does this help?
> >> >>
> >> >> Christian
> >> >>
> >> >>
> >> >>
> >> >> On Thu, Aug 17, 2017 at 9:21 PM, Tim Thompson <timat...@gmail.com>
> >> >> wrote:
> >> >> > Using the web:response-header() function from the BaseX Web Module
> >> >> > seems
> >> >> > to
> >> >> > work for overriding default headers
> >> >> > (http://docs.basex.org/wiki/Web_Module#web:response-header). I do
> >> >> > wonder
> >> >> > why
> >> >> > the behavior differs between this function and directly using a
> >> >> > rest:response element. Shouldn't these two be equivalent?
> >> >> >
> >> >> > Thanks,
> >> >> > Tim
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Tim A. Thompson
> >> >> > Discovery Metadata Librarian
> >> >> > Yale University Library
> >> >> >
> >> >> >
> >> >> > On Thu, Aug 17, 2017 at 11:50 AM, Tim Thompson <timat...@gmail.com
> >
> >> >> > wrote:
> >> >> >>
> >> >> >> Hello,
> >> >> >>
> >> >> >> Is it possible to overwrite default HTTP headers in RESTXQ? I am
> >> >> >> running
> >> >> >> BaseX in Tomcat 7 and have set the authmethod param set to
> "Custom."
> >> >> >> I
> >> >> >> wanted to define my own WWW-Authenticate header for Digest
> >> >> >> authentication. I
> >> >> >> tried something like this:
> >> >> >>
> >> >> >>  >> >> >> name="WWW-Authenticate"
> >> >> >> value="{
> >> >> >>   ``[Digest realm="BaseX",
> >> >> >> qop="auth, auth-int",
> >> >> >> algorithm=MD5,
> >> >> >> nonce=`{hash:md5(random:uuid())}`]``
> >> >> >> }"/>
> >> >> >>
> >> >> >> (in a custom response, as specified in
> >> >> >> http://docs.basex.org/wiki/RESTXQ#Custom_Response).
> >> >> >>
> >> >> >> However, the WWW-Authenticate header only returns a value of
> Custom
> >> >> >> realm="BaseX" no matter what I do in the custom RESTXQ response.
> >> >> >>
> >> >> >> Thanks in advance,
> >> >> >> Tim
> >> >> >>
> >> >> >> --
> >> >> >> Tim A. Thompson
> >> >> >> Discovery Metadata Librarian
> >> >> >> Yale University Library
> >> >> >>
> >> >> >
> >> >
> >> >
> >
> >
>

Re: [basex-talk] RESTXQ custom response

[Tim Thompson]

Right, the status can be set when using the http:response element directly,
but the web:response-header function does not seem to provide access to
that. It only allows one to set new headers, not set the status of the
response.




--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Fri, Aug 18, 2017 at 9:23 AM, Christian Grün <christian.gr...@gmail.com>
wrote:

> Hi Tim,
>
> That should be possible as well:
>
>   
> 
>   
>
> Cheers,
> Christian
>
>
>
> On Fri, Aug 18, 2017 at 3:20 PM, Tim Thompson <timat...@gmail.com> wrote:
> > Hi, Christian,
> >
> > Yes, it does. The only issue is that there doesn't seem to be a way to
> set
> > the HTTP status and message via web:response-header, is that right? This
> > would be a great feature to have :)
> >
> > Tim
> >
> > --
> > Tim A. Thompson
> > Discovery Metadata Librarian
> > Yale University Library
> >
> >
> > On Fri, Aug 18, 2017 at 4:25 AM, Christian Grün <
> christian.gr...@gmail.com>
> > wrote:
> >>
> >> Hi Tim,
> >>
> >> I am glad to hear the response header output does its job.
> >>
> >> As the function does nothing else than creating a response header
> >> (unfortunately with this standard caching directive – I think I will
> >> remove it along with BaseX 8.7), you could try to replace your
> >> response elements with the function’s result. Does this help?
> >>
> >> Christian
> >>
> >>
> >>
> >> On Thu, Aug 17, 2017 at 9:21 PM, Tim Thompson <timat...@gmail.com>
> wrote:
> >> > Using the web:response-header() function from the BaseX Web Module
> seems
> >> > to
> >> > work for overriding default headers
> >> > (http://docs.basex.org/wiki/Web_Module#web:response-header). I do
> wonder
> >> > why
> >> > the behavior differs between this function and directly using a
> >> > rest:response element. Shouldn't these two be equivalent?
> >> >
> >> > Thanks,
> >> > Tim
> >> >
> >> >
> >> > --
> >> > Tim A. Thompson
> >> > Discovery Metadata Librarian
> >> > Yale University Library
> >> >
> >> >
> >> > On Thu, Aug 17, 2017 at 11:50 AM, Tim Thompson <timat...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hello,
> >> >>
> >> >> Is it possible to overwrite default HTTP headers in RESTXQ? I am
> >> >> running
> >> >> BaseX in Tomcat 7 and have set the authmethod param set to "Custom."
> I
> >> >> wanted to define my own WWW-Authenticate header for Digest
> >> >> authentication. I
> >> >> tried something like this:
> >> >>
> >> >>  >> >> name="WWW-Authenticate"
> >> >> value="{
> >> >>   ``[Digest realm="BaseX",
> >> >> qop="auth, auth-int",
> >> >> algorithm=MD5,
> >> >> nonce=`{hash:md5(random:uuid())}`]``
> >> >> }"/>
> >> >>
> >> >> (in a custom response, as specified in
> >> >> http://docs.basex.org/wiki/RESTXQ#Custom_Response).
> >> >>
> >> >> However, the WWW-Authenticate header only returns a value of Custom
> >> >> realm="BaseX" no matter what I do in the custom RESTXQ response.
> >> >>
> >> >> Thanks in advance,
> >> >> Tim
> >> >>
> >> >> --
> >> >> Tim A. Thompson
> >> >> Discovery Metadata Librarian
> >> >> Yale University Library
> >> >>
> >> >
> >
> >
>

Re: [basex-talk] RESTXQ custom response

[Tim Thompson]

Hi, Christian,

Yes, it does. The only issue is that there doesn't seem to be a way to set
the HTTP status and message via web:response-header, is that right? This
would be a great feature to have :)

Tim

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Fri, Aug 18, 2017 at 4:25 AM, Christian Grün <christian.gr...@gmail.com>
wrote:

> Hi Tim,
>
> I am glad to hear the response header output does its job.
>
> As the function does nothing else than creating a response header
> (unfortunately with this standard caching directive – I think I will
> remove it along with BaseX 8.7), you could try to replace your
> response elements with the function’s result. Does this help?
>
> Christian
>
>
>
> On Thu, Aug 17, 2017 at 9:21 PM, Tim Thompson <timat...@gmail.com> wrote:
> > Using the web:response-header() function from the BaseX Web Module seems
> to
> > work for overriding default headers
> > (http://docs.basex.org/wiki/Web_Module#web:response-header). I do
> wonder why
> > the behavior differs between this function and directly using a
> > rest:response element. Shouldn't these two be equivalent?
> >
> > Thanks,
> > Tim
> >
> >
> > --
> > Tim A. Thompson
> > Discovery Metadata Librarian
> > Yale University Library
> >
> >
> > On Thu, Aug 17, 2017 at 11:50 AM, Tim Thompson <timat...@gmail.com>
> wrote:
> >>
> >> Hello,
> >>
> >> Is it possible to overwrite default HTTP headers in RESTXQ? I am running
> >> BaseX in Tomcat 7 and have set the authmethod param set to "Custom." I
> >> wanted to define my own WWW-Authenticate header for Digest
> authentication. I
> >> tried something like this:
> >>
> >>  >> name="WWW-Authenticate"
> >> value="{
> >>   ``[Digest realm="BaseX",
> >> qop="auth, auth-int",
> >> algorithm=MD5,
> >> nonce=`{hash:md5(random:uuid())}`]``
> >> }"/>
> >>
> >> (in a custom response, as specified in
> >> http://docs.basex.org/wiki/RESTXQ#Custom_Response).
> >>
> >> However, the WWW-Authenticate header only returns a value of Custom
> >> realm="BaseX" no matter what I do in the custom RESTXQ response.
> >>
> >> Thanks in advance,
> >> Tim
> >>
> >> --
> >> Tim A. Thompson
> >> Discovery Metadata Librarian
> >> Yale University Library
> >>
> >
>

Re: [basex-talk] RESTXQ custom response

[Tim Thompson]

Using the web:response-header() function from the BaseX Web Module seems to
work for overriding default headers (
http://docs.basex.org/wiki/Web_Module#web:response-header). I do wonder why
the behavior differs between this function and directly using a
rest:response element. Shouldn't these two be equivalent?

Thanks,
Tim


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library


On Thu, Aug 17, 2017 at 11:50 AM, Tim Thompson <timat...@gmail.com> wrote:

> Hello,
>
> Is it possible to overwrite default HTTP headers in RESTXQ? I am running
> BaseX in Tomcat 7 and have set the authmethod param set to "Custom." I
> wanted to define my own WWW-Authenticate header for Digest
> authentication. I tried something like this:
>
>  name="WWW-Authenticate"
> value="{
>   ``[Digest realm="BaseX",
> qop="auth, auth-int",
> algorithm=MD5,
> nonce=`{hash:md5(random:uuid())}`]``
> }"/>
>
> (in a custom response, as specified in http://docs.basex.org/wiki/
> RESTXQ#Custom_Response).
>
> However, the WWW-Authenticate header only returns a value of Custom
> realm="BaseX" no matter what I do in the custom RESTXQ response.
>
> Thanks in advance,
> Tim
>
> --
> Tim A. Thompson
> Discovery Metadata Librarian
> Yale University Library
>
>

[basex-talk] RESTXQ custom response

[Tim Thompson]

Hello,

Is it possible to overwrite default HTTP headers in RESTXQ? I am running
BaseX in Tomcat 7 and have set the authmethod param set to "Custom." I
wanted to define my own WWW-Authenticate header for Digest authentication.
I tried something like this:



(in a custom response, as specified in
http://docs.basex.org/wiki/RESTXQ#Custom_Response).

However, the WWW-Authenticate header only returns a value of Custom
realm="BaseX" no matter what I do in the custom RESTXQ response.

Thanks in advance,
Tim

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

Re: [basex-talk] Multipart form-data submissions from RESTXQ

[Tim Thompson]

Hi, Christian,

Thanks very much for looking into this. If I use the OxGarage TEI web
service through the front-end client to upload a file (
http://www.tei-c.org/oxgarage/), here is how it sends the request payload
on the back end. Non-ASCII characters are replaced with octal escape
sequences.

 Encapsulated multipart part:  (text/xml)
Content-Disposition: form-data; name="fileToConvert";
filename="tei.xml"\r\n
Content-Type: text/xml\r\n\r\n
eXtensible Markup Language
http://www.tei-c.org/ns/1.0; xml:lang="en">



Multipart test



unknown


unknown











 Regression
Equation 


http://www.w3.org/1998/Math/MathML;>


 Y 

 ^ 

 = 


 \316\262 


 1 


 + 


 \316\262 


 2 




 X 


 2 


 + 
 \342\200\246 
 + 


 \316\262 


 i 




 X 


 i 









Boundary:
\r\n-10775069631632435281298450283\r\n



--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

www.linkedin.com/in/timathompson
t...@princeton.edu

On Sat, Mar 11, 2017 at 10:30 AM, Christian Grün 
wrote:

> Hi Tim,
>
> Finally some feedback on this issue.
>
> It turned out that I cannot provide an easy fix for the problem you
> encountered. Your observations have already summarized the problem,
> and you have also found out what is happening internally: Whenever a
> multi-part body contains non-ASCII data, the
> "Content-Transfer-Encoding:base64" header is added [1].
>
> I am now mostly wondering how non-ASCII characters should be
> transferred, if not encoded as base64. Do you have some idea how the
> request would need to look like for TEI-C to be parseable?
>
> Cheers,
> Christian
>
> [1] https://github.com/BaseXdb/basex/blob/master/basex-core/
> src/main/java/org/basex/util/http/HttpClient.java#L271
>
>
>
> > Content-Type: text/xml\r\n
> > Content-Transfer-Encoding: 

Re: [basex-talk] basexhttpstop hangs in 8.6.1

[Tim Thompson]

I've also noticed similar behavior in version 8.6 on Ubuntu 14.04. If I
start the server in one terminal window and then run the stop script in a
separate window, the server does stop, but the basexhttpstop script keeps
hanging.

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

www.linkedin.com/in/timathompson
t...@princeton.edu

On Tue, Feb 28, 2017 at 6:22 AM, Marco Lettere  wrote:

> No Christian, really nothing special. Everything used to work neatly until
> Saturday when we tried 8.6.1 by just unzipping and launching. Btw we use
> Java 1.8.0_121 on both win7 and Ubuntu 16.04.
> Latest news same happens on Mac Sierra with Java 1.8.0_121.
> We've seen that there have been some commits recently on that parte of the
> code but I don't have a connection in my PC now ti check it in depth. Sorry.
> Ciao,
> Marco.
>
> Il 28 feb 2017 10:49 AM, "Christian Grün"  ha
> scritto:
>
>> Hi Marco,
>>
>> Hm, so it seems that the file is not deleted by the shutdown process.
>> I cannot reproduce this with the zip distribution on both Window 10
>> and Ubuntu; have you changed anything in the configuration files?
>>
>> Christian
>>
>>
>>
>> On Tue, Feb 28, 2017 at 9:38 AM, Marco Lettere 
>> wrote:
>> > Hi Christian,
>> > -d doesn't add much info.
>> > But we've found out that the shutdown script hangs on a file
>> BasexHTTP8985
>> > in the tmp directory. This happens btw on Windows 7 too.
>> > If we delete the file, while basexhttpstop is blocked, then
>> basexhttpstop
>> > ends immediately and correctly. If we ctrl-c basexhttpstop then at the
>> next
>> > shutdown we get a FileAlreadyExists exception.
>> >
>> > Thanks for tour support.
>> > Marco.
>> >
>> > Il 28 feb 2017 12:05 AM, "Christian Grün" 
>> ha
>> > scritto:
>> >>
>> >> Hi Marco,
>> >>
>> >> Could you please report back to us what exactly is output on the two
>> >> shells? Ideally, please run both commands with the -d flag.
>> >>
>> >> Thanks in advance,
>> >> Christian
>> >>
>> >> PS: Does it work with 8.6, 8.5.3 or older versions?
>> >>
>> >>
>> >>
>> >> > I just downloaded Basex861.zip for Linux (Ubuntu 16.04 the host).
>> >> >
>> >> > Unizipped and started basexhttp. Everything ok.
>> >> >
>> >> > In another shell I launch basexhttpstop which correctly shuts down
>> the
>> >> > server started before. But basexhttpstop itself does not terminate.
>> >> >
>> >> > This is causing some headache in our scripts that launch and stop
>> >> > several
>> >> > instances.
>> >> >
>> >> > Any clue?
>> >> >
>> >> > Thanks a lot,
>> >> >
>> >> > Marco.
>> >> >
>>
>

[basex-talk] Multipart form-data submissions from RESTXQ

[Tim Thompson]

Hello,

I have been agonizing over a problem with a service I'm trying to call from
RESTXQ. The service (www.tei-c.org/oxgarage/) only accepts
multipart/form-data submissions. It provides a front-end client for
uploading files from the browser, which calls a back-end RESTful service to
do document conversion.

I've installed a local instance of the service (
https://github.com/TEIC/oxgarage) and have it running on Tomcat 7, along
with BaseX.

The problem arises when I try to submit documents with non-ASCII characters
from RESTXQ. Looking at the network traffic, I can see that if the document
contains only ASCII characters, the multipart submission body is not base64
encoded. For example:

Encapsulated multipart part:  (text/xml)
Content-Disposition: form-data; name='fileToConvert';
filename='homework.xml'\r\n
Content-Type: text/xml\r\n\r\n
eXtensible Markup Language
http://www.w3.org/1999/xhtml;>



Test



http://www.w3.org/1998/Math/MathML;>



2






However, if the document does contain non-ASCII characters (such as β),
BaseX sets the Content-Transfer-Encoding to "base64." This causes the
OxGarage service to fail because it thinks it is receiving an image file
rather than a textual document. For example:

Content-Type: text/xml\r\n
Content-Transfer-Encoding: base64\r\n\r\n
eXtensible Markup Language
[truncated]
PGh0bWwgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGh0bWwiPjxoZWFkPjxtZXRhLz48\r\ndGl0bGU+VGVzdDwvdGl0bGU+PC9oZWFkPjxib2R5PjxtYXRoIHhtbG5zPSJodHRwOi8vd3d3Lncz\r\nLm9yZy8xOTk4L01hdGgvTWF0aE1MIj48bXN1Yj48bWk+zrI8L21pPjxtbj5Ud288L21

Attached here is a basic test case to replicate the problem: an HTML page
with a form and the RESTXQ function that it calls.

I've tried setting a new header to specify Content-Transfer-Encoding as
"binary" instead of "base64," but it doesn't replace the default header. Is
there any way that the encoding could be controlled from RESTXQ?

Thanks in advance!

Tim

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

www.linkedin.com/in/timathompson
t...@princeton.edu
Title: Test multipart-post



Click to test multipart-post from RESTXQ: 


 
A small HTML doc containing the following MathML markup will be submitted by the server to an instance of the OxGarage service.


β
2







multipart-test.xqm
Description: Binary data

Re: [basex-talk] Images

[Tim Thompson]

Joseph,

This thread from the xml-dev list might be of interest to you:
http://lists.xml.org/archives/xml-dev/201612/msg4.html. It discusses
DFDL (Data Format Description Language), an XML schema used in parsing
binary files (http://dfdlschemas.github.io/).

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Mon, Jan 2, 2017 at 6:14 AM, meumapple  wrote:

> Hi Christian,
>
> I recently had to deal with a Python library allowing to convert a png
> image into a matrix composed of pixel tuples [[(255,255,255), ], ...] which
> can then be changed and then saved back into png. I found it easy to
> manipulate it (ie, change colors, dimensions, etc...), so I was thinking
> that something similar in XQuery would be helpful for those working with
> images.
>
> Cheers,
> J.
>
> Il giorno 02 gen 2017, alle ore 11:42, Christian Grün <
> christian.gr...@gmail.com> ha scritto:
>
> Hi Joseph,
>
> There is no existing library for accessing image data. I would
> recommend you to take advantage of one of the various Java libraries.
>
> What exactly would you like to do with the image information?
>
> Cheers,
> Christian
>
>
>
>
> > On Sun, Dec 25, 2016 at 6:11 PM, meumapple  wrote:
> > Hi All,
> >
> > Is there a way to access images in BaseX as sequences of pixels? Thanks.
> >
> > Joseph
>

Re: [basex-talk] Large text files

[Tim Thompson]

Hi, Neven,

Yes, that's what I'll probably end up doing. My real motive for trying to
process the file as text was to test a BaseX extension function for parsing
n-triples, using one of Gunther Rademacher's REx Parser Generator
parsers[1].

I added a Java module with the parser to BaseX, and it works very well with
small to medium sized files, generating a parse tree based on the EBNF for
the format. I was just curious to see how it would perform with very large
files.

Best,
Tim

[1] http://www.bottlecaps.de/rex/

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Fri, Nov 25, 2016 at 12:47 PM, Neven Jovanović <filologan...@gmail.com>
wrote:

> Hi Tim,
>
> may I suggest that you convert the n-triples file to RDF/XML format
> (I'm using Pythong rdflib for such tasks,
> <http://rdflib.readthedocs.io/en/stable/>)? Perhaps it would be easier
> for BaseX to ingest the XML instead of text format (which it thinks
> the n-triples are).
>
> Best,
>
> Neven
>
> Neven Jovanovic, Zagreb
>
> On 25 November 2016 at 18:21, Christian Grün <christian.gr...@gmail.com>
> wrote:
> > Hi Tim,
> >
> > In BaseX, texts/strings are internally represented as byte arrays. Due
> > to the 32 bit limitation of Java arrays, the file will be too large to
> > be oped as single text in main-memory.
> >
> > To be honest, I didn’t have a similar use case before, so I guess the
> > best solution for now will be to split the file into smaller chunks
> > before processing it with BaseX.
> >
> > Cheers,
> > Christian
> >
> >
> >
> > On Fri, Nov 25, 2016 at 6:07 PM, Tim Thompson <timat...@gmail.com>
> wrote:
> >> Hello,
> >>
> >> I have a large file[1] (3.5G unzipped) in the n-triples RDF format that
> I
> >> would like to work with in BaseX. When I try to read in the file using
> >> file:read-text(), I get the following error:
> >>
> >> Error:
> >> Version: BaseX 8.6 beta 8fa97ca
> >> Java: Oracle Corporation, 1.8.0_73
> >> OS: Linux, amd64
> >> Stack Trace:
> >> java.lang.NegativeArraySizeException
> >> at java.util.Arrays.copyOf(Arrays.java:3236)
> >> at org.basex.util.TokenBuilder.addByte(TokenBuilder.java:247)
> >> at org.basex.util.TokenBuilder.add(TokenBuilder.java:176)
> >> at org.basex.io.in.TextInput.cache(TextInput.java:143)
> >> at org.basex.io.in.TextInput.content(TextInput.java:132)
> >> at org.basex.query.value.item.StrStream.materialize(
> StrStream.java:71)
> >> at org.basex.query.value.item.StrStream.string(StrStream.java:44)
> >> at org.basex.query.expr.ParseExpr.toToken(ParseExpr.java:273)
> >> at org.basex.query.expr.ParseExpr.toEmptyToken(ParseExpr.java:261)
> >> at org.basex.query.func.fn.FnSubstring.item(FnSubstring.java:22)
> >> at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44)
> >> at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99)
> >> at org.basex.query.scope.MainModule$1.next(MainModule.java:122)
> >> at org.basex.query.QueryContext.cache(QueryContext.java:648)
> >> at org.basex.query.QueryProcessor.cache(QueryProcessor.java:116)
> >> at org.basex.core.cmd.AQuery.query(AQuery.java:87)
> >> at org.basex.core.cmd.XQuery.run(XQuery.java:22)
> >> at org.basex.core.Command.run(Command.java:255)
> >> at org.basex.core.Command.execute(Command.java:93)
> >> at org.basex.gui.GUI.exec(GUI.java:479)
> >> at org.basex.gui.GUI.access$3(GUI.java:433)
> >> at org.basex.gui.GUI$7.run(GUI.java:421)
> >>
> >> When I try to create a text database using the GUI, I get an error
> stating
> >> that the file could not be parsed.
> >>
> >> Is it possible to work with text files that are this large using BaseX?
> >>
> >> Thank you,
> >> Tim
> >>
> >> [1] Available for download here:
> >> http://www.bne.es/es/Inicio/Perfiles/Bibliotecarios/DatosEnlazados/
> DescargaFicheros/
> >> (http://datos.bne.es/datadumps/autoridades.nt.bz2)
> >>
> >> --
> >> Tim A. Thompson
> >> Metadata Librarian (Spanish/Portuguese Specialty)
> >> Princeton University Library
> >>
>

[basex-talk] Large text files

[Tim Thompson]

Hello,

I have a large file[1] (3.5G unzipped) in the n-triples RDF format that I
would like to work with in BaseX. When I try to read in the file using
file:read-text(), I get the following error:

Error:
Version: BaseX 8.6 beta 8fa97ca
Java: Oracle Corporation, 1.8.0_73
OS: Linux, amd64
Stack Trace:
java.lang.NegativeArraySizeException
at java.util.Arrays.copyOf(Arrays.java:3236)
at org.basex.util.TokenBuilder.addByte(TokenBuilder.java:247)
at org.basex.util.TokenBuilder.add(TokenBuilder.java:176)
at org.basex.io.in.TextInput.cache(TextInput.java:143)
at org.basex.io.in.TextInput.content(TextInput.java:132)
at org.basex.query.value.item.StrStream.materialize(StrStream.java:71)
at org.basex.query.value.item.StrStream.string(StrStream.java:44)
at org.basex.query.expr.ParseExpr.toToken(ParseExpr.java:273)
at org.basex.query.expr.ParseExpr.toEmptyToken(ParseExpr.java:261)
at org.basex.query.func.fn.FnSubstring.item(FnSubstring.java:22)
at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44)
at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99)
at org.basex.query.scope.MainModule$1.next(MainModule.java:122)
at org.basex.query.QueryContext.cache(QueryContext.java:648)
at org.basex.query.QueryProcessor.cache(QueryProcessor.java:116)
at org.basex.core.cmd.AQuery.query(AQuery.java:87)
at org.basex.core.cmd.XQuery.run(XQuery.java:22)
at org.basex.core.Command.run(Command.java:255)
at org.basex.core.Command.execute(Command.java:93)
at org.basex.gui.GUI.exec(GUI.java:479)
at org.basex.gui.GUI.access$3(GUI.java:433)
at org.basex.gui.GUI$7.run(GUI.java:421)

When I try to create a text database using the GUI, I get an error stating
that the file could not be parsed.

Is it possible to work with text files that are this large using BaseX?

Thank you,
Tim

[1] Available for download here:
http://www.bne.es/es/Inicio/Perfiles/Bibliotecarios/DatosEnlazados/DescargaFicheros/
(http://datos.bne.es/datadumps/autoridades.nt.bz2)

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

[basex-talk] Parsing XML within string constructors

[Tim Thompson]

Hello,

I have a string that I want to parse using fn:parse-xml-fragment (in BaseX
8.4.1). I noticed that there seemed to be a difference when wrapping the
string in a string constructor versus plain quotes.

Using the string constructor seems to require a second function call in
order to parse the XML.

The following three expressions illustrate the difference. Numbers 2 and 3
return parsed XML, whereas number 1 returns the original, unparsed string.
Is this the expected behavior when using string constructors?

1.
``[span start="" end="3"Sed/span ut perspiciatis]`` =>
parse-xml-fragment()

2.
``[span start="" end="3"Sed/span ut perspiciatis]`` =>
parse-xml-fragment() => parse-xml-fragment()

3.
'span start="" end="3"Sed/span ut perspiciatis' =>
parse-xml-fragment()


--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

[basex-talk] Lookups and arrows

[Tim Thompson]

Hello,

I'm testing some XQuery 3.1 features against a JSON-LD[1] document and had
a few questions. In the JSON-LD format, the "@" symbol has special
semantics in key names, but seems to cause problems with the 3.1 lookup
operator.

For example:

json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")

works as expected, but

json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld;)?@context

throws an error: [XPST0003] No specifier after lookup operator: '@'.

Also, when using the "=>" operator, should it be possible to perform a
lookup on the last expression in a chain, if that expression returns a map?

For example:

json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")
evaluates to "http://iiif.io/api/presentation/2/context.json;

and

json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") =>
json-doc()

returns another map object. So, how would one achieve this:

json-doc(json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld
")("@context"))("@context")

using the arrow operator?

Thanks,
Tim


[1] http://www.w3.org/TR/json-ld/


--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

Re: [basex-talk] Lookups and arrows

[Tim Thompson]

Thanks, Joe! I guess I'd glossed over the "funky" example ;-) Regarding the
arrow operator, I was wondering whether something like this was possible:

json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") =>
json-doc()("@context")

which throws an error: [XPST0003] Unexpected end of query: '("@context")'.

Tim

On Sat, Jan 9, 2016 at 3:53 PM, Joe Wicentowski <joe...@gmail.com> wrote:

> Hi Tim,
>
> For your first question, I think your example falls into what the spec
> calls "funky looking" keys.  See the 3rd bullet point example under
> http://www.w3.org/TR/xquery-31/#id-lookup:
>
> > funky /  @string"), an appropriate lookup for a map with rather odd conventions for
> keys.
>
> In other words, I think you're stuck with the ("@context") approach.
>
> As to your second question, it looks like what you're proposing should
> work - but am I reading you as saying you get an error with your proposed
> approach?  If so, what's the error?
>
> Joe
>
> Sent from my iPad
>
> _
> From: Tim Thompson <timat...@gmail.com>
> Sent: Saturday, January 9, 2016 3:11 PM
> Subject: [basex-talk] Lookups and arrows
> To: BaseX <basex-talk@mailman.uni-konstanz.de>
>
>
>
> Hello,
>
> I'm testing some XQuery 3.1 features against a JSON-LD[1] document and had
> a few questions. In the JSON-LD format, the "@" symbol has special
> semantics in key names, but seems to cause problems with the 3.1 lookup
> operator.
>
> For example:
>
> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")
>
> works as expected, but
>
> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)?@context
>
> throws an error: [XPST0003] No specifier after lookup operator: '@'.
>
> Also, when using the "=>" operator, should it be possible to perform a
> lookup on the last expression in a chain, if that expression returns a map?
>
> For example:
>
> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")
> evaluates to " http://iiif.io/api/presentation/2/context.json;
>
> and
>
> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") =>
> json-doc()
>
> returns another map object. So, how would one achieve this:
>
> json-doc(json-doc(" 
> http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context"))("@context")
>
>
> using the arrow operator?
>
> Thanks,
> Tim
>
>
> [1] http://www.w3.org/TR/json-ld/
>
>
> --
> Tim A. Thompson
> Metadata Librarian (Spanish/Portuguese Specialty)
> Princeton University Library
>
>
>
>

[basex-talk] Question about namespace prefix bindings

[Tim Thompson]

Hello,

I have a small database of library catalog records that I am trying to run
some simple queries on, but I'm running into a namespace problem.

Some docs in the database have the following structure, with a wrapper
collection element and two record elements:

http://www.loc.gov/MARC21/slim; xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="
http://www.loc.gov/MARC21/slim
http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd;>
  
...
  
  
...
  


But others lack a wrapper collection and have only a single record element,
which also lacks a namespace declaration, like so:


...


So, I run the following query (using the DBA query interface):

declare namespace marc = "http://www.loc.gov/MARC21/slim;;

let $m := collection("marc-data")/marc:record[@type = "Bibliographic"]
return count($m)

And the count returns 0. However, if I change the predicate in the let
clause to:

let $m := collection("marc-data")/*[local-name() = "record"][@type =
"Bibliographic"]

the count returns 45.

If I am declaring the namespace in my query prolog and associating it with
the same prefix as the one used in my XML documents, why don't I get the
desired prefix/namespace mapping? I'm probably missing something basic, but
this didn't immediately make sense to me.

Thank you,
Tim


--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

Re: [basex-talk] Question about namespace prefix bindings

[Tim Thompson]

Sorry, never mind. I see that the documents that lack are namespace
declarations are just not well formed. Actually, I'm a little surprised
that it was possible to save them to the DB in the first place!

Thanks,
Tim

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Wed, Jan 6, 2016 at 12:32 PM, Tim Thompson <timat...@gmail.com> wrote:

> Hello,
>
> I have a small database of library catalog records that I am trying to run
> some simple queries on, but I'm running into a namespace problem.
>
> Some docs in the database have the following structure, with a wrapper
> collection element and two record elements:
>
> http://www.loc.gov/MARC21/slim; xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="
> http://www.loc.gov/MARC21/slim
> http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd;>
>   
> ...
>   
>   
> ...
>   
> 
>
> But others lack a wrapper collection and have only a single record
> element, which also lacks a namespace declaration, like so:
>
> 
> ...
> 
>
> So, I run the following query (using the DBA query interface):
>
> declare namespace marc = "http://www.loc.gov/MARC21/slim;;
>
> let $m := collection("marc-data")/marc:record[@type = "Bibliographic"]
> return count($m)
>
> And the count returns 0. However, if I change the predicate in the let
> clause to:
>
> let $m := collection("marc-data")/*[local-name() = "record"][@type =
> "Bibliographic"]
>
> the count returns 45.
>
> If I am declaring the namespace in my query prolog and associating it with
> the same prefix as the one used in my XML documents, why don't I get the
> desired prefix/namespace mapping? I'm probably missing something basic, but
> this didn't immediately make sense to me.
>
> Thank you,
> Tim
>
>
> --
> Tim A. Thompson
> Metadata Librarian (Spanish/Portuguese Specialty)
> Princeton University Library
>
>

Re: [basex-talk] Which zip did BaseX choke on?

[Tim Thompson]

I may be wrong, but I think you need to wrap the $err:code in db:output():
see http://docs.basex.org/wiki/XQuery_Update#Returning_Results

Tim

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Fri, Oct 16, 2015 at 10:28 AM, Hondros, Constantine (ELS-AMS) <
c.hond...@elsevier.com> wrote:

> Hmm, it is surprisingly hard to get some useful logging information out of
> this.
>
>
>
> Here for example I fall foul of a XQuery syntax here: “all expressions
> must be updating or return an empty sequence. ”
>
>
>
> for $zip in file:list($hm)
>
> return  try {
>
> db:add("myDB", $hm || '\' || $zip, "", map {
> 'intparse': true() })
>
> } catch * {
>
> $err:code
>
> }
>
>
>
> Can anyone help me rephrase this so when BaseX halts on a bad zipfile, I
> at least know what the name of the zipfile is?
>
>
>
> C.
>
>
>
>
>
> *From:* Alexander Holupirek [mailto:a...@holupirek.de]
> *Sent:* 16 October 2015 15:23
> *To:* Hondros, Constantine (ELS-AMS)
> *Cc:* BaseX
> *Subject:* Re: [basex-talk] Which zip did BaseX choke on?
>
>
>
> Have you tried using try/catch?
>
>
>
> for zip in zips return try { unzip() } catch ...
>
>
> Am 16.10.2015 um 15:16 schrieb Hondros, Constantine (ELS-AMS) <
> c.hond...@elsevier.com>:
>
> I love BaseX for the simplicity it brings to XML handling. But this is a
> problem I have not encountered before.
>
>
>
> I am creating a DB from about 17,000 small zipfiles, each containing a
> directory structure and somewhere within each, some XML. BaseX chokes on
> one of these files giving the error:  “invalid entry size (expected 0 but
> got 11083 bytes)”.
>
>
>
> So clearly one or more of the zips is invalid -  but which one(s)?
>
>
>
> Is there any way that BaseX can echo to me the file that is causing the
> error? DEBUG is set to TRUE, but I don’t get any morer verbose output. I am
> using the GUI, and running a BSX script too.
>
>
>
> Thanks in advance,
>
> C.
>
>
> --
>
> Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> Netherlands, Registration No. 33156677, Registered in The Netherlands.
>
>
> --
>
> Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> Netherlands, Registration No. 33156677, Registered in The Netherlands.
>

[basex-talk] BaseX, Zorba, e-mail module

[Tim Thompson]

Hello,

I would like to be able to send e-mail from a Web app I am developing in
BaseX and was wondering whether anyone else had implemented this
functionality (or whether a BaseX e-mail module might be in the works?).

I see that Zorba has an e-mail module with support for SMTP and IMAP (not
to mention many other useful modules), which leads me to wonder whether
anyone has worked on a BaseX connector for Zorba.

In the meantime, I decided to try using the BaseX process module to call
Zorba, like so:

proc:system(
  'zorba', (
'-q', convert:binary-to-string(
  db:retrieve(
'out', 'email.xq'
  )
)
  )
)

I stored the XQuery code for Zorba as a raw file in the BaseX database,
since storing it in the webapp directory caused an error, due to the
unrecognized Zorba module declarations. This approach seems to work
adequately, but does anyone have any suggestions for improvement, or a
different approach?

Thanks in advance,
Tim


--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

[basex-talk] Converting to UTF-8 in SQL module

[Tim Thompson]

Hello,

I am using the BaseX SQL module to query the Oracle database of a library
catalog. Its character data is encoded as ISO 2709 (MARC-8)[1] and is
stored in Oracle as US7ASCII.

The data contains diacritics with combining characters like this:
http://www.fileformat.info/info/unicode/char/0301/index.htm

When I run the query in BaseX, all characters with combining accents are
output as something like Ì#x81;

Has anyone had experience handling this kind of encoding issue in BaseX, or
does anyone have any solutions/approaches to recommend as a way to convert
this data to UTF-8?

Thanks in advance,
Tim

[1] https://en.wikipedia.org/wiki/ISO_2709
[2] http://www.fileformat.info/info/unicode/char/00e8/index.htm

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library t...@princeton.edu

Re: [basex-talk] BaseX subcollections

[Tim Thompson]

Thanks, Dirk. Basically, I just want to create hierarchical collections
within a database, with some subcollections being possible empty. But if I
do CREATE DB parent, I'm not able to follow that with something like, CREATE
DB parent/child. Is this possible by other means? (Sorry for the newbie
question; I'm still getting familiarized with how things work in BaseX.)

Tim

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Thu, Aug 13, 2015 at 6:07 AM, Dirk Kirsten d...@basex.org wrote:

 Hello Tim,

 I am not sure I understand your question completely. Basically every way
 how you can add a document to a database you can specify a path, e.g. using
 the XQuery database module or by using the CREATE document to create a new
 database and initialize it. Is there anything specific you are looking for?

 Also, please not that BaseX 6.3.1 is _very_ outdated... I don't think you
 can find much valuable information there except looking for the history of
 BaseX.

 Cheers
 Dirk


 On 08/12/2015 07:15 PM, Tim Thompson wrote:

 Hello,

 I see that I can create subcollections or paths within a database if I
 create the database then use the ADD TO command to add a resource at a
 specified path. Is this the only way to create hiearchical structure within
 a database? I couldn't find many details about this in the documentation.

 I see that the changelog for BaseX 6.3.1 includes, [ADD] REST: support
 for hierarchical collections added; details will follow, but I don't see
 anything about this in the wiki page for the REST module.

 Thanks in advance,
 Tim

 --
 Tim A. Thompson
 Metadata Librarian (Spanish/Portuguese Specialty)
 Princeton University Library


 --
 Dirk Kirsten, BaseX GmbH, http://basexgmbh.de
 |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz
 |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer:
 |   Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle
 `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22

[basex-talk] BaseX subcollections

[Tim Thompson]

Hello,

I see that I can create subcollections or paths within a database if I
create the database then use the ADD TO command to add a resource at a
specified path. Is this the only way to create hiearchical structure within
a database? I couldn't find many details about this in the documentation.

I see that the changelog for BaseX 6.3.1 includes, [ADD] REST: support for
hierarchical collections added; details will follow, but I don't see
anything about this in the wiki page for the REST module.

Thanks in advance,
Tim

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

Re: [basex-talk] Performance with joins?

[Tim Thompson]

Adding a text() step to the predicate does dramatically reduce the
execution time.

This query:

declare namespace marc=http://www.loc.gov/MARC21/slim;;

for $m in collection(
  latin_hold_20150730
)/marc:collection/marc:record,
$r in collection(
  latin_hold_20150730
)/root/row[BIB_ID/text() = $m/marc:controlfield[@tag = '001']]
group by $key := $r/ITEM_ID

return
test n={
  $key
}{
$m/marc:datafield[@tag = '245']/marc:subfield[@code = 'a']/text()
}/test

Executes in 257.13 ms. But it shouldn't be necessary to explicitly specify
the text() node, right (or would even be bad practice to do so[1])?

Thanks,
Tim

[1]
http://stackoverflow.com/questions/12970162/xquery-join-query#comment-17592189


--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Tue, Aug 4, 2015 at 8:25 AM, Tim Thompson timat...@gmail.com wrote:

 I was mistaken; I hadn't enabled indexes on this database. However, after
 creating attribute and text indexes, the query actually seems to take
 longer to execute (483622.95 ms on last run), although the query plan
 itself doesn't seem to have changed:

 Compiling:
 - pre-evaluating collection(latin_hold_20150730)
 - pre-evaluating collection(latin_hold_20150730)
 Query:
 declare namespace marc=http://www.loc.gov/MARC21/slim;; for $m in
 collection( latin_hold_20150730 )/marc:collection/marc:record, $r in
 collection( latin_hold_20150730 )/root/row[BIB_ID =
 $m/marc:controlfield[@tag = '001']] group by $key := $r/ITEM_ID return
 test n={ $key }{ $m/marc:datafield[@tag = '245']/marc:subfield[@code =
 'a']/string() }/test
 Optimized Query:
 for $m_0 in (db:open-pre(latin_hold_20150730,0),
 ...)/marc:collection/marc:record for $r_1 in
 (db:open-pre(latin_hold_20150730,0), ...)/root/row[(BIB_ID =
 $m_0/marc:controlfield[(@tag = 001)])] let (: post-group :) $m_4 := $m_0
 group by $key_2 := $r_1/ITEM_ID return element test { (attribute n {
 ($key_2) }, $m_4/marc:datafield[(@tag = 245)]/marc:subfield[(@code =
 a)]/string()) }
 Result:
 - Hit(s): 7587 Items
 - Updated: 0 Items
 - Printed: 505 KB
 - Read Locking: local [latin_hold_20150730]
 - Write Locking: none
 Timing:
 - Parsing: 1.04 ms
 - Compiling: 0.86 ms
 - Evaluating: 483604.85 ms
 - Printing: 16.2 ms
 - Total Time: 483622.95 ms

 Query plan:
 QueryPlan compiled=true
   GFLWOR
 For
   Var name=$m id=0/
   IterPath
 DBNodeSeq size=2
   DBNode name=latin_hold_20150730 pre=0/
   DBNode name=latin_hold_20150730 pre=365692/
 /DBNodeSeq
 IterStep axis=child test=marc:collection/
 IterStep axis=child test=marc:record/
   /IterPath
 /For
 For
   Var name=$r id=1/
   IterPath
 DBNodeSeq size=2
   DBNode name=latin_hold_20150730 pre=0/
   DBNode name=latin_hold_20150730 pre=365692/
 /DBNodeSeq
 IterStep axis=child test=root/
 IterStep axis=child test=row
   CmpG op==
 CachedPath
   IterStep axis=child test=BIB_ID/
 /CachedPath
 IterPath
   VarRef
 Var name=$m id=0/
   /VarRef
   IterStep axis=child test=marc:controlfield
 CmpG op==
   CachedPath
 IterStep axis=attribute test=tag/
   /CachedPath
   Str value=001 type=xs:string/
 /CmpG
   /IterStep
 /IterPath
   /CmpG
 /IterStep
   /IterPath
 /For
 GroupBy
   Spec
 Var name=$key id=2/
 IterPath
   VarRef
 Var name=$r id=1/
   /VarRef
   IterStep axis=child test=ITEM_ID/
 /IterPath
   /Spec
 /GroupBy
 CElem
   QNm value=test type=xs:QName/
   CAttr
 QNm value=n type=xs:QName/
 VarRef
   Var name=$key id=2/
 /VarRef
   /CAttr
   MixedPath
 VarRef
   Var name=$m id=4/
 /VarRef
 IterStep axis=child test=marc:datafield
   CmpG op==
 CachedPath
   IterStep axis=attribute test=tag/
 /CachedPath
 Str value=245 type=xs:string/
   /CmpG
 /IterStep
 IterStep axis=child test=marc:subfield
   CmpG op==
 CachedPath
   IterStep axis=attribute test=code/
 /CachedPath
 Str value=a type=xs:string/
   /CmpG
 /IterStep
 FnString name=string([item])/
   /MixedPath
 /CElem
   /GFLWOR
 /QueryPlan


 --
 Tim A. Thompson
 Metadata Librarian (Spanish/Portuguese Specialty)
 Princeton University Library


 On Tue, Aug 4, 2015 at 3:23 AM, Christian Grün christian.gr...@gmail.com
 wrote:

 Dear Tim,

 The query plan indicates that no index is applied. Your query may be
 evaluated faster when rewriting BIB_ID to BIB_ID/text(). I will
 see if this can automatically be done by the query

Re: [basex-talk] Performance with joins?

[Tim Thompson]

I was mistaken; I hadn't enabled indexes on this database. However, after
creating attribute and text indexes, the query actually seems to take
longer to execute (483622.95 ms on last run), although the query plan
itself doesn't seem to have changed:

Compiling:
- pre-evaluating collection(latin_hold_20150730)
- pre-evaluating collection(latin_hold_20150730)
Query:
declare namespace marc=http://www.loc.gov/MARC21/slim;; for $m in
collection( latin_hold_20150730 )/marc:collection/marc:record, $r in
collection( latin_hold_20150730 )/root/row[BIB_ID =
$m/marc:controlfield[@tag = '001']] group by $key := $r/ITEM_ID return
test n={ $key }{ $m/marc:datafield[@tag = '245']/marc:subfield[@code =
'a']/string() }/test
Optimized Query:
for $m_0 in (db:open-pre(latin_hold_20150730,0),
...)/marc:collection/marc:record for $r_1 in
(db:open-pre(latin_hold_20150730,0), ...)/root/row[(BIB_ID =
$m_0/marc:controlfield[(@tag = 001)])] let (: post-group :) $m_4 := $m_0
group by $key_2 := $r_1/ITEM_ID return element test { (attribute n {
($key_2) }, $m_4/marc:datafield[(@tag = 245)]/marc:subfield[(@code =
a)]/string()) }
Result:
- Hit(s): 7587 Items
- Updated: 0 Items
- Printed: 505 KB
- Read Locking: local [latin_hold_20150730]
- Write Locking: none
Timing:
- Parsing: 1.04 ms
- Compiling: 0.86 ms
- Evaluating: 483604.85 ms
- Printing: 16.2 ms
- Total Time: 483622.95 ms
Query plan:
QueryPlan compiled=true
  GFLWOR
For
  Var name=$m id=0/
  IterPath
DBNodeSeq size=2
  DBNode name=latin_hold_20150730 pre=0/
  DBNode name=latin_hold_20150730 pre=365692/
/DBNodeSeq
IterStep axis=child test=marc:collection/
IterStep axis=child test=marc:record/
  /IterPath
/For
For
  Var name=$r id=1/
  IterPath
DBNodeSeq size=2
  DBNode name=latin_hold_20150730 pre=0/
  DBNode name=latin_hold_20150730 pre=365692/
/DBNodeSeq
IterStep axis=child test=root/
IterStep axis=child test=row
  CmpG op==
CachedPath
  IterStep axis=child test=BIB_ID/
/CachedPath
IterPath
  VarRef
Var name=$m id=0/
  /VarRef
  IterStep axis=child test=marc:controlfield
CmpG op==
  CachedPath
IterStep axis=attribute test=tag/
  /CachedPath
  Str value=001 type=xs:string/
/CmpG
  /IterStep
/IterPath
  /CmpG
/IterStep
  /IterPath
/For
GroupBy
  Spec
Var name=$key id=2/
IterPath
  VarRef
Var name=$r id=1/
  /VarRef
  IterStep axis=child test=ITEM_ID/
/IterPath
  /Spec
/GroupBy
CElem
  QNm value=test type=xs:QName/
  CAttr
QNm value=n type=xs:QName/
VarRef
  Var name=$key id=2/
/VarRef
  /CAttr
  MixedPath
VarRef
  Var name=$m id=4/
/VarRef
IterStep axis=child test=marc:datafield
  CmpG op==
CachedPath
  IterStep axis=attribute test=tag/
/CachedPath
Str value=245 type=xs:string/
  /CmpG
/IterStep
IterStep axis=child test=marc:subfield
  CmpG op==
CachedPath
  IterStep axis=attribute test=code/
/CachedPath
Str value=a type=xs:string/
  /CmpG
/IterStep
FnString name=string([item])/
  /MixedPath
/CElem
  /GFLWOR
/QueryPlan


--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Tue, Aug 4, 2015 at 3:23 AM, Christian Grün christian.gr...@gmail.com
wrote:

 Dear Tim,

 The query plan indicates that no index is applied. Your query may be
 evaluated faster when rewriting BIB_ID to BIB_ID/text(). I will
 see if this can automatically be done by the query compiler.

 Best,
 Christian


 On Tue, Aug 4, 2015 at 4:42 AM, Tim Thompson timat...@gmail.com wrote:
  Hello,
 
  I have a database containing two resources/documents: they both represent
  the same set of library catalog records (7728 records in each), but
 they
  each contain different data that I want to join.
 
  The first resource looks like this:
 
  marc:collection
  xmlns:marc=http://www.loc.gov/MARC21/slim;
  marc:record
  marc:leader01225cam a2200373Mi 4500/marc:leader
  marc:controlfield tag=0015323084/marc:controlfield
  marc:datafield
  ind1=1
  ind2=4
  tag=245
  marc:subfield code=aEls teleclubs a les illes Balears
  :/marc:subfield
  /marc:datafield
  /marc:record
  marc:record
  marc:leader01225cam a2200373Mi 4500/marc:leader
  marc:controlfield tag=0015323084/marc:controlfield

Re: [basex-talk] Parsing options on db:add (cdata-section-elements)

[Tim Thompson]

Thank you, that works perfectly! I also found a workaround using html as
the output method, which lets me create a CDATA section with
syntax-specific comments, for example:

script type=text/turtle{concat('#![CDATA[', $body, '#]]')}/script

(Following the W3C example for embedding Turtle in XHTML[1].)

Best regards,
Tim

[1] http://www.w3.org/TR/turtle/#xhtml

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Mon, Jul 13, 2015 at 4:40 AM, Christian Grün christian.gr...@gmail.com
wrote:

 Hi Tim,

 I must confess I haven't run your full example, but you'll probably
 need to explicitly specify the namespace of your script element, as
 shown in the following two function calls:

   file:write(result.xml,
 xml xmlns=http://www.w3.org/1999/xhtml;
   script lt; /script
 /xml,
 output:serialization-parameters
   output:method
 value=xhtml/
   output:cdata-section-elements
 value=Q{{http://www.w3.org/1999/xhtml}}script/
 /output:serialization-parameters
   )

   file:write(result.xml,
 xml xmlns=http://www.w3.org/1999/xhtml;
   script lt; /script
 /xml,
 map {
   'cdata-section-elements':
   'Q{http://www.w3.org/1999/xhtml}script'
 }
   )

 Otherwise, the CDATA section does only apply to the empty namespace:

   file:write(result.xml,
 xml xmlns=http://www.w3.org/1999/xhtml;
   script xmlns= lt; /script
 /xml,
 map { 'cdata-section-elements': 'script' }
   )

 Regarding your second question:

   (: Does this work? :)
   %output:cdata-section-elements(script)

 Yes, that's indeed a valid way to specify serialization parameters.

 Hope this helps,
 Christian

Re: [basex-talk] Parsing options on db:add (cdata-section-elements)

[Tim Thompson]

Dirk,

Thanks for your reply. Here's a simple example. When I run this query with
Saxon HE 9.6, the output contains a CDATA section, but when I run it in the
BaseX GUI, there is no CDATA and the angle brackets are escaped. Has this
serialization parameter been implemented in BaseX?

xquery version 3.0;

declare copy-namespaces no-preserve, no-inherit;
declare default element namespace http://www.w3.org/1999/xhtml;;
declare namespace output = http://www.w3.org/2010/xslt-xquery-serialization
;
declare option output:method xhtml;
declare option output:indent yes;
declare option output:encoding utf-8;
declare option output:cdata-section-elements script;

let $turtle :=
turtle
![CDATA[
@prefix ex: http://example.org/ .
@prefix test: http://test.org/ .

ex:test1 a test:Test .
]]
/turtle

let $html :=
html
head
meta
content=text/html; charset=UTF-8
http-equiv=Content-Type/
titleTurtle Test/title
/head
body
h1Turtle Test/h1
script
type=text/turtle{
$turtle/text()
}/script
/body
/html

return
$html


Saxon output:

?xml version=1.0 encoding=utf-8?
html
xmlns=http://www.w3.org/1999/xhtml;
head
meta
content=text/html; charset=utf-8
http-equiv=Content-Type /
titleTurtle Test/title
/head
body
h1Turtle Test/h1
script type=text/turtle![CDATA[

@prefix ex: http://example.org/ .
@prefix test: http://test.org/ .

ex:test1 a test:Test .

]]/script
/body
/html


BaseX output:

html
  xmlns=http://www.w3.org/1999/xhtml;
  head
meta
  content=text/html; charset=UTF-8
  http-equiv=Content-Type /
titleTurtle Test/title
  /head
  body
h1Turtle Test/h1
script type=text/turtle

@prefix ex: lt;http://example.org/gt; .
@prefix test: lt;http://test.org/gt; .

ex:test1 a test:Test .

/script
  /body
/html



--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Fri, Jul 10, 2015 at 4:39 AM, Dirk Kirsten d...@basex.org wrote:

 Hello Tim,

 CDATA is just ordinary (encoded) text to XQuery, i.e. there is no
 special CDATA data type. Therefore, instead of saving the document in
 this way you should adapt the output accordingly. So when you serialize
 your output you can use the map {cdata-section-elements: script} to
 output the result as CDATA section. You can read a bit more about this
 in our documentation at http://docs.basex.org/wiki/Serialization

 Cheers
 Dirk

 On 07/09/2015 08:31 PM, Tim Thompson wrote:
  Hello,
 
  I am trying to retrieve some non-XML RDF data (as text/turtle) from
  a SPARQL endpoint. I need to output the Turtle as a CDATA section
  inside a script tag in an XForms (X)HTML file, then add that file to
  a database.
 
  The file gets added correctly, but I'm not able to generate the CDATA
  section. Here is the function:
 
  db:add(xforms, document {
  $xslt-pi,
  $css,
  $form
  }, turtle-test.xml, map {cdata-section-elements: script})
 
  When I run this, I get an error: [bxerr:BASX0002] Unknown database
  option 'cdata-section-elements'.
 
  I get the same error no matter which option I use (not just
  cdata-section-elements), so I guess I may be using the wrong syntax.
 
  Hope someone can point me in the right direction!
 
  Thanks in advance,
  Tim
 
  --
  Tim A. Thompson
  Metadata Librarian (Spanish/Portuguese Specialty)
  Princeton University Library
 

 --
 Dirk Kirsten, BaseX GmbH, http://basexgmbh.de
 |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz
 |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer:
 |   Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle
 `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22

Re: [basex-talk] Parsing options on db:add (cdata-section-elements)

[Tim Thompson]

Excellent! Thank you for fixing this so quickly.

Tim

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library
t...@princeton.edu

On Fri, Jul 10, 2015 at 9:16 AM, Christian Grün christian.gr...@gmail.com
wrote:

 Hi Tim,

 The default element namespace declaration will now be considered when
 parsing the cdata-section-elements option. I have just uploaded a new
 snapshot [1], and 8.2.3 will available soon.

 Thanks to your feedback, two more bugs in the W3 XQuery 3.1 Test Suite
 could be fixed. In includes around 28.000 tests, and we are currently
 covering around 99%.

 Thanks,
 Christian

 [1] http://files.basex.org/releases/latest/



 On Fri, Jul 10, 2015 at 2:08 PM, Christian Grün
 christian.gr...@gmail.com wrote:
  Hi Tim,
 
  Thanks for the example code. It noticed that BaseX fails to recognize
  CDATA section elements if a default element namespace is set:
 
declare default element namespace http://www.w3.org/1999/xhtml;;
declare namespace output = 
 http://www.w3.org/2010/xslt-xquery-serialization;;
declare option output:cdata-section-elements xml;
xml![CDATA[  ]]/xml
 
  The following query works (but it's quite obviously not what you need):
 
declare namespace output = 
 http://www.w3.org/2010/xslt-xquery-serialization;;
declare option output:cdata-section-elements xml;
xml![CDATA[  ]]/xml
 
  I will check the serialization spec again and fix this as soon as
 possible.
  Christian
 
 
  On Fri, Jul 10, 2015 at 1:40 PM, Tim Thompson timat...@gmail.com
 wrote:
  Dirk,
 
  Thanks for your reply. Here's a simple example. When I run this query
 with
  Saxon HE 9.6, the output contains a CDATA section, but when I run it in
 the
  BaseX GUI, there is no CDATA and the angle brackets are escaped. Has
 this
  serialization parameter been implemented in BaseX?
 
  xquery version 3.0;
 
  declare copy-namespaces no-preserve, no-inherit;
  declare default element namespace http://www.w3.org/1999/xhtml;;
  declare namespace output =
  http://www.w3.org/2010/xslt-xquery-serialization;;
  declare option output:method xhtml;
  declare option output:indent yes;
  declare option output:encoding utf-8;
  declare option output:cdata-section-elements script;
 
  let $turtle :=
  turtle
  ![CDATA[
  @prefix ex: http://example.org/ .
  @prefix test: http://test.org/ .
 
  ex:test1 a test:Test .
  ]]
  /turtle
 
  let $html :=
  html
  head
  meta
  content=text/html; charset=UTF-8
  http-equiv=Content-Type/
  titleTurtle Test/title
  /head
  body
  h1Turtle Test/h1
  script
  type=text/turtle{
  $turtle/text()
  }/script
  /body
  /html
 
  return
  $html
 
 
  Saxon output:
 
  ?xml version=1.0 encoding=utf-8?
  html
  xmlns=http://www.w3.org/1999/xhtml;
  head
  meta
  content=text/html; charset=utf-8
  http-equiv=Content-Type /
  titleTurtle Test/title
  /head
  body
  h1Turtle Test/h1
  script type=text/turtle![CDATA[
 
  @prefix ex: http://example.org/ .
  @prefix test: http://test.org/ .
 
  ex:test1 a test:Test .
 
  ]]/script
  /body
  /html
 
 
  BaseX output:
 
  html
xmlns=http://www.w3.org/1999/xhtml;
head
  meta
content=text/html; charset=UTF-8
http-equiv=Content-Type /
  titleTurtle Test/title
/head
body
  h1Turtle Test/h1
  script type=text/turtle
 
  @prefix ex: lt;http://example.org/gt; .
  @prefix test: lt;http://test.org/gt; .
 
  ex:test1 a test:Test .
 
  /script
/body
  /html
 
 
 
  --
  Tim A. Thompson
  Metadata Librarian (Spanish/Portuguese Specialty)
  Princeton University Library
 
 
  On Fri, Jul 10, 2015 at 4:39 AM, Dirk Kirsten d...@basex.org wrote:
 
  Hello Tim,
 
  CDATA is just ordinary (encoded) text to XQuery, i.e. there is no
  special CDATA data type. Therefore, instead of saving the document in
  this way you should adapt the output accordingly. So when you serialize
  your output you can use the map {cdata-section-elements: script} to
  output the result as CDATA section. You can read a bit more about this
  in our documentation at http://docs.basex.org/wiki/Serialization
 
  Cheers
  Dirk
 
  On 07/09/2015 08:31 PM, Tim Thompson wrote:
   Hello,
  
   I am trying to retrieve some non-XML RDF data (as text/turtle) from
   a SPARQL endpoint. I need to output the Turtle as a CDATA section
   inside a script tag in an XForms (X)HTML file, then add that file
 to
   a database.
  
   The file gets added correctly, but I'm not able to generate the CDATA
   section. Here is the function:
  
   db:add(xforms, document {
   $xslt-pi,
   $css,
   $form
   }, turtle-test.xml, map {cdata-section-elements: script})
  
   When I run this, I get an error: [bxerr:BASX0002] Unknown database
   option 'cdata-section-elements

Re: [basex-talk] Parsing options on db:add (cdata-section-elements)

[Tim Thompson]

Hello again,

Just to follow up on my original inquiry, I am trying to use the
cdata-section-elements serialization parameter to write an XHTML XForms
document (now to the filesystem rather than a database).

The BaseX documentation indicates that the file:write function can take a
$params argument with serialization parameters, and it and gives
cdata-section-elements as an example. However, I haven't been successful in
my attempt to use this.

Here is a Gist that illustrates my basic use case:
https://gist.github.com/Timathom/486b1fcf05d2c69ec176

I submit to a RESTXQ endpoint from an XForms document, then write another
XForms doc to the filesystem, placing the results of a SPARQL query in the
script element.

When I open the resulting turtle-test.xhtml file, the script element
still lacks a CDATA section and the angle brackets have been escaped.

Thanks again for your help!

Tim

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library


On Fri, Jul 10, 2015 at 9:24 AM, Tim Thompson timat...@gmail.com wrote:

 Excellent! Thank you for fixing this so quickly.

 Tim

 --
 Tim A. Thompson
 Metadata Librarian (Spanish/Portuguese Specialty)
 Princeton University Library
 t...@princeton.edu

 On Fri, Jul 10, 2015 at 9:16 AM, Christian Grün christian.gr...@gmail.com
  wrote:

 Hi Tim,

 The default element namespace declaration will now be considered when
 parsing the cdata-section-elements option. I have just uploaded a new
 snapshot [1], and 8.2.3 will available soon.

 Thanks to your feedback, two more bugs in the W3 XQuery 3.1 Test Suite
 could be fixed. In includes around 28.000 tests, and we are currently
 covering around 99%.

 Thanks,
 Christian

 [1] http://files.basex.org/releases/latest/



 On Fri, Jul 10, 2015 at 2:08 PM, Christian Grün
 christian.gr...@gmail.com wrote:
  Hi Tim,
 
  Thanks for the example code. It noticed that BaseX fails to recognize
  CDATA section elements if a default element namespace is set:
 
declare default element namespace http://www.w3.org/1999/xhtml;;
declare namespace output = 
 http://www.w3.org/2010/xslt-xquery-serialization;;
declare option output:cdata-section-elements xml;
xml![CDATA[  ]]/xml
 
  The following query works (but it's quite obviously not what you need):
 
declare namespace output = 
 http://www.w3.org/2010/xslt-xquery-serialization;;
declare option output:cdata-section-elements xml;
xml![CDATA[  ]]/xml
 
  I will check the serialization spec again and fix this as soon as
 possible.
  Christian
 
 
  On Fri, Jul 10, 2015 at 1:40 PM, Tim Thompson timat...@gmail.com
 wrote:
  Dirk,
 
  Thanks for your reply. Here's a simple example. When I run this query
 with
  Saxon HE 9.6, the output contains a CDATA section, but when I run it
 in the
  BaseX GUI, there is no CDATA and the angle brackets are escaped. Has
 this
  serialization parameter been implemented in BaseX?
 
  xquery version 3.0;
 
  declare copy-namespaces no-preserve, no-inherit;
  declare default element namespace http://www.w3.org/1999/xhtml;;
  declare namespace output =
  http://www.w3.org/2010/xslt-xquery-serialization;;
  declare option output:method xhtml;
  declare option output:indent yes;
  declare option output:encoding utf-8;
  declare option output:cdata-section-elements script;
 
  let $turtle :=
  turtle
  ![CDATA[
  @prefix ex: http://example.org/ .
  @prefix test: http://test.org/ .
 
  ex:test1 a test:Test .
  ]]
  /turtle
 
  let $html :=
  html
  head
  meta
  content=text/html; charset=UTF-8
  http-equiv=Content-Type/
  titleTurtle Test/title
  /head
  body
  h1Turtle Test/h1
  script
  type=text/turtle{
  $turtle/text()
  }/script
  /body
  /html
 
  return
  $html
 
 
  Saxon output:
 
  ?xml version=1.0 encoding=utf-8?
  html
  xmlns=http://www.w3.org/1999/xhtml;
  head
  meta
  content=text/html; charset=utf-8
  http-equiv=Content-Type /
  titleTurtle Test/title
  /head
  body
  h1Turtle Test/h1
  script type=text/turtle![CDATA[
 
  @prefix ex: http://example.org/ .
  @prefix test: http://test.org/ .
 
  ex:test1 a test:Test .
 
  ]]/script
  /body
  /html
 
 
  BaseX output:
 
  html
xmlns=http://www.w3.org/1999/xhtml;
head
  meta
content=text/html; charset=UTF-8
http-equiv=Content-Type /
  titleTurtle Test/title
/head
body
  h1Turtle Test/h1
  script type=text/turtle
 
  @prefix ex: lt;http://example.org/gt; .
  @prefix test: lt;http://test.org/gt; .
 
  ex:test1 a test:Test .
 
  /script
/body
  /html
 
 
 
  --
  Tim A. Thompson
  Metadata Librarian (Spanish/Portuguese Specialty)
  Princeton University Library
 
 
  On Fri, Jul 10, 2015 at 4:39 AM, Dirk Kirsten d...@basex.org wrote:
 
  Hello Tim,
 
  CDATA is just

[basex-talk] Parsing options on db:add (cdata-section-elements)

[Tim Thompson]

Hello,

I am trying to retrieve some non-XML RDF data (as text/turtle) from a
SPARQL endpoint. I need to output the Turtle as a CDATA section inside a
script tag in an XForms (X)HTML file, then add that file to a database.

The file gets added correctly, but I'm not able to generate the CDATA
section. Here is the function:

db:add(xforms, document {
$xslt-pi,
$css,
$form
}, turtle-test.xml, map {cdata-section-elements: script})

When I run this, I get an error: [bxerr:BASX0002] Unknown database option
'cdata-section-elements'.

I get the same error no matter which option I use (not just
cdata-section-elements), so I guess I may be using the wrong syntax.

Hope someone can point me in the right direction!

Thanks in advance,
Tim

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

[basex-talk] DBA login error

[Tim Thompson]

Hello,

I'm a first-time BaseX user and have just downloaded version 8.2.2.

When I start an HTTP server and attempt to log in to the DBA app with
default admin credentials, I get an HTTP 500 error stating that there was a
Problem accessing /dba/login-check. However, if I go back and refresh the
login page, I see that I have been logged in as admin and can access the
DBA interface. Similarly, I get a 500 error when trying to access
/dba/logout.

Is there something I have failed to configure?

Thanks in advance,
Tim

Stack trace for /dba/login-check error:

Version: BaseX 8.2.2
Java: Oracle Corporation, 1.7.0_60
OS: Mac OS X, x86_64
Stack Trace:
java.lang.ArrayIndexOutOfBoundsException: 1
at org.basex.query.func.admin.AdminWriteLog.item(AdminWriteLog.java:25)
at org.basex.query.expr.ParseExpr.value(ParseExpr.java:64)
at org.basex.query.QueryContext.value(QueryContext.java:421)
at org.basex.query.expr.List.value(List.java:148)
at org.basex.query.QueryContext.value(QueryContext.java:421)
at org.basex.query.func.StaticFunc.invValue(StaticFunc.java:192)
at org.basex.query.func.FuncCall.invoke(FuncCall.java:87)
at org.basex.query.func.FuncCall.value(FuncCall.java:60)
at org.basex.query.QueryContext.value(QueryContext.java:421)
at org.basex.query.expr.If.value(If.java:133)
at org.basex.query.QueryContext.value(QueryContext.java:421)
at org.basex.query.expr.If.value(If.java:133)
at org.basex.query.expr.gflwor.GFLWOR.value(GFLWOR.java:72)
at org.basex.query.expr.TypeCheck.value(TypeCheck.java:127)
at org.basex.query.QueryContext.value(QueryContext.java:421)
at org.basex.query.expr.If.value(If.java:133)
at org.basex.query.QueryContext.value(QueryContext.java:421)
at org.basex.query.func.StaticFunc.invValue(StaticFunc.java:192)
at org.basex.query.func.FuncCall.invoke(FuncCall.java:87)
at org.basex.query.func.FuncCall.value(FuncCall.java:60)
at org.basex.query.func.FuncCall.iter(FuncCall.java:66)
at org.basex.query.MainModule.iter(MainModule.java:110)
at org.basex.query.QueryContext.iter(QueryContext.java:349)
at org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:55)
at org.basex.http.restxq.RestXqModule.process(RestXqModule.java:101)
at org.basex.http.restxq.RestXqFunction.process(RestXqFunction.java:109)
at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:44)
at org.basex.http.BaseXServlet.service(BaseXServlet.java:64)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at 
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at 
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
at 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at 
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)


--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library