Re: [basex-talk] Dr. Mike Kay's proposals for XPath and XQuery 4.0

2020-11-16 Thread Marco Lettere
Couple of things directly out of my mind even if more library than core 
language oriented are YAML support and the new proposal for convenient 
HTTP client functions. I remember a proposal by you, Christian, together 
with Adam maybe?


M.

On 16/11/20 15:30, Loren Cahlander wrote:

Please make sure to keep me in the loop.  I will need to keep xqDoc up to date.


On Nov 16, 2020, at 6:28 AM, Christian Grün  wrote:


I don't know how many implementers we have on this list, outside of Christian 
and co., but thought you all might find it interesting.

Everyone: raise your hands and get involved ;)

Michael’s initiative to push for a new version of XQuery, XPath & co.
is very promising indeed. If you are interested in sharing your ideas
which features you would like to see in a new version: Now may still
be a good time for it!




PS I found this announcement on the XML.com Slack channel, fyi.

[1] Mike Kay's post from Slack:
I've been working on fleshing out my ideas for XSLT/XPath 4.0, based on my XML 
Prague paper earlier this year. I'm hoping to gather together a community group 
of some kind to take this forward; meanwhile I've published a very preliminary 
set of drafts:
XSLT: https://www.saxonica.com/qt4specs/XT/Overview-diff.html
F: https://www.saxonica.com/qt4specs/FO/Overview-diff.html
XPath: https://www.saxonica.com/qt4specs/XP/xpath-40-diff.html
XQuery: https://www.saxonica.com/qt4specs/XQ/xquery-40-diff.html
Preparing for the rotten tomatoes: let me know what you think and whether you 
want to participate.
NOTE: each document has a summary of changes at the end. It's probably best to 
go there first.





Re: [basex-talk] BaseX GUI and remote editing

2020-11-13 Thread Marco Lettere

Thanks for the hint Bridger.
We would like to stay lightweight thus I'd prefer to not introduce big 
IDEs only for pushing code.
So for the moment (at least for me having Linux) I'll use Nautilus 
(Gnome file browser) to connect to the remote folder containing the code 
and opening files locally with BaseX Gui...  we have to be much more 
careful with coworking and conflict management. But with no build 
process (thanks XQ) and developing in rather isolated sections (thanks 
microservices) may be it could work.

We'll see...
Thanks again.
M.

On 13/11/20 17:33, Bridger Dyson-Smith wrote:

Hi Marco,

I'm sure others will have better thoughts on this, so please take this 
with a grain of salt. Jetbrains IntelliJ (and other IDEs) provide 
Remote Deployment options, and I would think Eclipse would offer 
something similar. I.e. write locally, and then push changes to the 
remote server with a keyboard chord. However, these types of 
connections require SSH or something similar.


I don't know if that's a helpful thought or not.
Best,
Bridger

PS Shameless plug from a happy user: the Jetbrains IDEs can leverage 
Reece Dunn's awesome XQuery plugin.


On Fri, Nov 13, 2020 at 9:50 AM Marco Lettere <mailto:m.lett...@gmail.com>> wrote:


Hello all!

We are in the slightly uncomfortable condition to do some remote
coding.
Means write XQ code that has to be kept on a server different from
our PCs.

We don't want to slow down our very agile development workflow by
passing everytime from a Git repo ...

Has anyone here some experience with remote editing of this kind?
Using
BaseX Gui is out of discussion, isn't it?

Thanks,

Marco.





[basex-talk] BaseX GUI and remote editing

2020-11-13 Thread Marco Lettere

Hello all!

We are in the slightly uncomfortable condition to do some remote coding. 
Means write XQ code that has to be kept on a server different from our PCs.


We don't want to slow down our very agile development workflow by 
passing everytime from a Git repo ...


Has anyone here some experience with remote editing of this kind? Using 
BaseX Gui is out of discussion, isn't it?


Thanks,

Marco.



Re: [basex-talk] Paging results of an XQuery-Search

2020-09-25 Thread Marco Lettere
Xquery native windowing instructions despite a slightly complex syntax 
are extremely powerfull and you can rely on the optimizer being able to 
optimize where ever possible.

M.

https://www.w3.org/TR/xquery-30/#id-windows

On 25/09/20 15:55, Omar Siam wrote:

Perhaps you want to use fn:subsequence($hits, $start, $num)?
But of course if you have millions of hits that is not efficient enough.

Am 25.09.2020 um 13:23 schrieb Günter Dunz-Wolff:

Hi all,
is it possible to present the results of an XQuery search in parts of 
50? In order to speed up the search, only the first 50 results should 
be displayed and only if the user is interested in further results, 
50 new results each should be displayed. Is this even possible? I did 
not find anything about it in the documentation.


Example:
let $collection := collection("data")
let $hits := $collection//*:s[.//text() contains text {$query_string}]
let $count_hits := count($hits)
return

{
if $count_hits <= 50 then

{for $hit in $hits
return
{$hit}

else
???
}


Thanks for any advice.
Best regards
Guenter








Re: [basex-talk] Accessing Flexmark (Java) from Query

2020-09-21 Thread Marco Lettere

On 20/09/20 22:42, Andreas Mixich wrote:

Currently I get the error:

   [XPTY0004] com.vladsch.flexmark.parser.Parser$Builder:build: Multiple
functions with 1.


Hi Andreas,

this is because of Java distinguishes method signatures not only by 
number of parameters but also by type. You have to use the middle-dot 
notation to clearly define which method you need as in [1].


Regards,

Marco.

[1] https://docs.basex.org/wiki/Java_Bindings#Functions_and_Variables



Re: [basex-talk] missing web.xml from docker image

2020-09-15 Thread Marco Lettere

On 15/09/20 10:35, Andy Bunce wrote:
... as BaseX seems to run surprisingly well on ARM64 machines such as 
the Odroid N2 and Raspberry Pi 4.


Absolutely!

I have been running a demo/development site of a customer's web 
application (fully developed in BaseX) on my home Raspberry PI 3 for 
weeks. :-)


Until the cloud service provider of choice resolved some issues they had 
on their side ;-).


M.



Re: [basex-talk] Yaml serialization

2020-08-12 Thread Marco Lettere




@Marco: Do you know which conversions are provided by FasterXML:
arbitrary XML → YAML, arbitrary YAML → XML, or both? If the underlying
conversion rules are promising, we could include them a 'fasterxml'
format in BaseX (similar to the 'jsonml' format).

As far as I can understand from the docs and the code on github there 
Jackson is a sort of hub with several dataformat modules which are able 
to parse and serialize from the various formats.


I don't know whether, with the term arbitrary, you mean to be able to 
provide your own syntax for the XML serialization. In this case I am not 
sure.


M.



Re: [basex-talk] Yaml serialization

2020-08-12 Thread Marco Lettere
Hm, not sure but maybe FasterXML's Jackson? It supports alternative non 
JSON data formats among which YAML, XML.

But maybe that one is a huge dependency.
M.

On 12/08/20 12:38, Christian Grün wrote:

Hi Marco, hi Fabrice,

YAML serialization would be a nice feature indeed.

Are you aware of existing (ideally bidirectional) mappings that would 
be appropriate for general use?


Cheers
Christian





Marco Lettere mailto:m.lett...@gmail.com>> 
schrieb am Mi., 12. Aug. 2020, 12:04:


Dear BaseX team,

just out of curiosity, could you imagine what kind of effort it
would be
to provide YAML as another serialization method for BaseX in
addition to
XML, HTML, JSON, CSV?

The datamodel seems pretty close to the JSON one so maybe a very
similar
transformation to XML could be implemented ...

Ciao,

Marco.





[basex-talk] Yaml serialization

2020-08-12 Thread Marco Lettere

Dear BaseX team,

just out of curiosity, could you imagine what kind of effort it would be 
to provide YAML as another serialization method for BaseX in addition to 
XML, HTML, JSON, CSV?


The datamodel seems pretty close to the JSON one so maybe a very similar 
transformation to XML could be implemented ...


Ciao,

Marco.



Re: [basex-talk] Transaction Support

2020-07-17 Thread Marco Lettere
I'm not the maximum expert on this because I'm only a user. So maybe 
Christian could correct me.
The PUL is actually an implementation detail. It's not something you 
access explicitly. Thus no. There is no explicit mechanism.
The behaviour (START TRANS, do smthg, COMMIT) is implicit in each and 
every XQuery script that the compiler recognizes to be "updating" 
(a.k.a. performing database modifications).
Even if it accesses multiple databases. So to me this looks as being 
transactional in nature.
For instance when I call a RestXQ service which stores a document in one 
DB and then updates a record in another one and this second operation 
raises an error causing the RestXQ to quit then both the operations will 
be canceled automatically. This is very different from, for example, 
MongoDB.
Of course this means that there are complex locking strategies which you 
have to pay attention to when performing particularly complex operations 
but this is another point.


I hoffe alles goht guet i de Schwiz (hm ... Schwitzerduetsch is hard to 
write)

M.

On 16/07/20 17:55, Reto Peter wrote:


Hi Marco

Thanks for answering

But that means Basex is NOT supporting database transactions

Or is it possible to implement real transactions (START TRANS, do 
something, COMMIT trans) with that PUL or something?


Reto

*From:*BaseX-Talk [mailto:basex-talk-boun...@mailman.uni-konstanz.de] 
*On Behalf Of *Marco Lettere

*Sent:* 16 July 2020 17:46
*To:* basex-talk@mailman.uni-konstanz.de
*Subject:* Re: [basex-talk] Transaction Support

Hi Reto,

AFAIK Basex is transactional in the sense that whenever you start a 
sequence of commands or an XQuery script, all the "updating 
operations" that modify the database are always stored in a PUL (a 
list of potential updates).


Only when the script terminates all the operations on the DB are 
effectively committed.


There is no explicit rollback operation.

Regards,

Marco.

On 16/07/20 17:40, Reto Peter wrote:

I am evaluating BaseX for my XML project.

I need transaction support like

-Start transaction

-Run queries (read, write, update)

-Commit or rollback transaction

When I see the documentation, it lists Transaction Manager. But
when I look at the details, I cannot find anything like that.

Anyone can explain me how is the support, or is there an add-on or
planned something?

Best regards

Reto, Frauenfeld, Schweiz





Re: [basex-talk] Transaction Support

2020-07-16 Thread Marco Lettere

Hi Reto,
AFAIK Basex is transactional in the sense that whenever you start a 
sequence of commands or an XQuery script, all the "updating operations" 
that modify the database are always stored in a PUL (a list of potential 
updates).
Only when the script terminates all the operations on the DB are 
effectively committed.

There is no explicit rollback operation.
Regards,
Marco.

On 16/07/20 17:40, Reto Peter wrote:


I am evaluating BaseX for my XML project.

I need transaction support like

-Start transaction

-Run queries (read, write, update)

-Commit or rollback transaction

When I see the documentation, it lists Transaction Manager. But when I 
look at the details, I cannot find anything like that.


Anyone can explain me how is the support, or is there an add-on or 
planned something?


Best regards

Reto, Frauenfeld, Schweiz





Re: [basex-talk] BaseX 9.4: The Summer Edition

2020-07-14 Thread Marco Lettere
Even in this strange 2020 Summertime can now definitely begin with the
Summer edition which looks awesome!!!
Thanks to all of you!
Marco.

Il mar 14 lug 2020, 13:40 Christian Grün  ha
scritto:

> We are glad to give you Version 9.4 of BaseX, our XML framework,
> database system and XQuery 3.1 processor:
>
>   https://basex.org/
>
> We have focused on rewriting and optimizing complex XQuery code and
> speeding up your RESTXQ apps:
>
> WEB APPLICATIONS
> - DBA: support for millions of log entries
> - LOGTRACE option: write trace output to logs or STDERR
> - rest:init: Full or partial initialization of RESTXQ cache
> - basexhttp: register job services
>
> GUI
> - result view: show number of results and result size
> - Shift F4-F7: toggle search options (case, regeex, ..)
> - Ctrl-Shift-B: jump to matching bracket
>
> XQUERY LOCKING
> - single lock option for reading and updating queries
> - Java bindings: annotation added for updating functions
>
> XQUERY FUNCTIONS
> - convenience functions: json:doc, csv:doc, html:doc
> - element names: convert:encode-key, convert:decode-key
> - jobs:eval: option added for writing log entries
> - session module: only create new sessions if required
>
> XQUERY PERFORMANCE
> - boolean comparisons: flatten nested expressions
> - boolean expressions: merge conjunctions
> - comparisons: simplify operands with if expression
> - database functions: always open at compile time
> - databases: propagate to more expressions at compile time
> - filters: inline context for single items
> - filters: rewrite to simple maps
> - FLWOR: inline where clauses into let clauses
> - FLWOR: merge last with return clause, rewrite to simple map
> - FLWOR: rewrite "return if ..." to "where ... return"
> - fn:for-each, fn:filter: rewrite to FLWORs and filters
> - if expression, EBV tests: simplify boolean tests
> - lists, sets, logical expressions: flatten expressions
> - lists: rewrite to union expressions
> - logical expressions: apply more boolean algebra rules
> - paths: remove redundant predicates
> - predicates with name functions: rewrite to name tests
> - predicates: merge expressions, discard redundant tests
> - predicates: remove of superfluous and redundant tests
> - set expressions: merge operands
> - simple map: inline cast expressions
> - simple maps: faster evaluation
> - simple maps: merge operands
> - simple maps: rewrite to paths, optimize for index access
> - switch: rewrite to if expression
> - types: skip redundant checks, promotions  and conversions
>
> For a more comprehensive list of added and updated features, check out
> our documentation (docs.basex.org) and check out the GitHub issues
> (github.com/BaseXdb/basex/issues).
>
> Have fun,
> Your BaseX Team
>


Re: [basex-talk] Websites that use BaseX

2020-06-24 Thread Marco Lettere

Hi Jonathan,
we as a company have used Nubisware for several "front-facing" 
applications. They are not exactly ecommerce but full blown enterprise 
applications with complex UIs and/or based on APIs.


 * A rating service (with dedicated UI) based on contract ontologies;
 * a system for remote realtime monitoring and failure prediction in
   industrial settings with a websocket push based interface [1,2]
 * An API and viewer/simulator for IHE XDW based clinical workflow
   tracking integrated with the Camunda BPMN Engine
 * A Psychological Patient Record with web based frontend
 * A frontend plus service orchestrator for the daily management of a
   Coworking site.
 * ...

We are very happy of the results and our customers too (which is much 
more important). What our customers like the most, about XQuery/BaseX 
based solutions, is the very low footprint and impact on their 
infrastructures and deployments.

M.

[1] 
https://www.linkedin.com/posts/global-solutions-%26-services_shortendistances-savecosts-customersatisfaction-activity-6678200998059683840-bedJ 

[2] 
https://www.linkedin.com/posts/global-solutions-%26-services_shortendistances-savecosts-customersatisfaction-activity-6673859515550715905-21GS


On 23/06/20 22:56, Liam R. E. Quin wrote:

On Tue, 2020-06-23 at 16:34 -0400, Joe Wicentowski wrote:

Hi all,

I'd welcome any and all contributions to my list of XQuery-powered
resources:

   https://github.com/joewiz/xquery-power

Not a major ecommerce site :) but...

Most Web pages on From Old Books _dot_ Org -
https://www.fromoldbooks.org/ - incorporate some content generated by
XQuery, and many of the pages are entirely generated at runtime by
XQuery, using BaseX.

There may still be some pages using qizx/open.

Please don't add this to your site, but, It's possible to add, ;show-
query=text/plain to the pages under /Search/ to see the XQuery itself.

Liam





[basex-talk] distinct-values and type coercion

2020-05-19 Thread Marco Lettere

Dear all,

after installing Basex 9.3.3 one application of ours started failing in 
a particular functionality.


We tracked it down to the following example code:

let $in :=
  
    
Object/20383920-5bb1-4397-8b51-b752863acddf
    
  
let $s as xs:string* := distinct-values($in/source/reference/string())
return $s

This code now returns the following error: [XPTY0004] Cannot treat 
xs:untypedAtomic* as xs:string*: 
distinct-values($in_0/element(source)/element(reference)).
Whereas before it was automatically casting to xs:string now it seems 
that the type coercion in the let clause is too restrictive.


This happens only with the distinct-values applied by the way.

Is it correct now or was it before? For the moment we solved by removing 
the type enforcement in the let clause.


Thank you for the clarification.

Regards,

Marco.



Re: [basex-talk] BaseX 9.3.3: Query optimizations, Bug Fixes

2020-05-15 Thread Marco Lettere

Once more: THANKS!
M.

On 15/05/20 11:35, Christian Grün wrote:

Dear users and supporters of BaseX,

We have just released a new maintenance version of our XML framework,
database and XQuery processor:

http://basex.org

Apart from minor bug fixes, and in preparation to BaseX 9.4, we have
added even more query optimizations, FLWOR, path and predicate
rewritings. Feel free to check out the full list of added rewritings
and enhancements on Github:

   https://github.com/BaseXdb/basex/issues?q=is%3Aissue+is%3Aclosed

Have fun with the new release!

Christian
BaseX Team

PS: With BaseX 10 (no release date yet), we’ll switch from Java 8 to
11, the current long-term-support release of Java.





Re: [basex-talk] Performance loss between version 9.2.4 and 9.3.2 when executing specific xQuery

2020-05-08 Thread Marco Lettere
And even more curiously, we are also working in this very same time on 
handling fetches from OAI-PMH sources! :-D

M.

On 08/05/20 13:37, Imsieke, Gerrit, le-tex wrote:
Just saying that I find it sooo interesting to learn at which places 
and for which purposes BaseX is being employed. Have a nice weekend!


On 08.05.2020 13:31, BIRKNER Michael wrote:

Hi Christian,


thank you for your answers. As you can guess the queries I sent in my 
original email are just simplified  examples.



The real XML structure is like the following (its library data in 
format "MarcXML", here you see an example: 
https://www.loc.gov/standards/marcxml/Sandburg/sandburg.xml)




… … …



Mag. Michael Birkner
AK Wien - Bibliothek
1040, Prinz Eugen Straße 20-22
T: +43 1 501 65 12455
F: +43 1 501 65 142455
M: +43 664 88957669





Re: [basex-talk] A question about RESTXQ rest:form-param()

2020-04-23 Thread Marco Lettere

I set up this restxq [1].
And call it with something like [2].
Got an output of [3] which looks like what I expect.
M.

[1]
module namespace t = "urn:test";

declare
  %rest:path("test")
  %rest:POST
  %rest:consumes("application/x-www-form-urlencoded")
  %output:method("html")
function t:test() {
 
   
 {request:parameter-names() ! (. || "=" || 
request:parameter(.))}

   
 
};

[2]
curl --location --request POST 'http://localhost:8984/test' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'param1=Hello' \
--data-urlencode 'param2=World'

[3]


param1=Hello param2=World



On 22/04/20 16:03, Zimmel, Daniel wrote:

the mime type seems correct, all params are being sent. I do not have problems 
getting the body, only request:parameter-names() does not give any values. 
(request:header works, for example)

-----Ursprüngliche Nachricht-
Von: Marco Lettere 
Gesendet: Mittwoch, 22. April 2020 15:31
An: Zimmel, Daniel ; basex-talk@mailman.uni-konstanz.de
Betreff: Re: AW: [basex-talk] A question about RESTXQ rest:form-param()

Are you setting the mime type to "application/x-www-form-urlencoded"?
This is the prerequisite in order to get the body of a post parsed to form 
parameters.
M.

On 22/04/20 15:22, Zimmel, Daniel wrote:

Thanks for the hint Marco,

I just tried request:parameter-names(): that one works with GET but with POST I 
get an empty-sequence, I am not sure what the problem is here.
I can access the full (unparsed) body, though (I can use that).

Daniel

-Ursprüngliche Nachricht-
Von: Marco Lettere 
Gesendet: Mittwoch, 22. April 2020 14:24
An: basex-talk@mailman.uni-konstanz.de
Betreff: Re: [basex-talk] A question about RESTXQ rest:form-param()

Hi Daniel,
another way is to use request:* functions [1] available for programmatic 
inspection of the request.
Cheers,
M.

[1] https://docs.basex.org/wiki/Request_Module


On 22/04/20 12:18, Zimmel, Daniel wrote:

Hi Daniel,

RTFM. Simply pass the body in %rest:POST("{$body}").

Sorry for disturbing!

Daniel

-Ursprüngliche Nachricht-
Von: Zimmel, Daniel 
Gesendet: Mittwoch, 22. April 2020 11:27
An: BaseX 
Betreff: [basex-talk] A question about RESTXQ rest:form-param()

Hi,

in an HTML form with a RESTXQ POST action, do I really have to declare 
%rest:form-param() for every param, or is there a way to pass the full query 
component in one parameter?

Best, Daniel





Re: [basex-talk] A question about RESTXQ rest:form-param()

2020-04-22 Thread Marco Lettere

Are you setting the mime type to "application/x-www-form-urlencoded"?
This is the prerequisite in order to get the body of a post parsed to 
form parameters.

M.

On 22/04/20 15:22, Zimmel, Daniel wrote:

Thanks for the hint Marco,

I just tried request:parameter-names(): that one works with GET but with POST I 
get an empty-sequence, I am not sure what the problem is here.
I can access the full (unparsed) body, though (I can use that).

Daniel

-Ursprüngliche Nachricht-
Von: Marco Lettere 
Gesendet: Mittwoch, 22. April 2020 14:24
An: basex-talk@mailman.uni-konstanz.de
Betreff: Re: [basex-talk] A question about RESTXQ rest:form-param()

Hi Daniel,
another way is to use request:* functions [1] available for programmatic 
inspection of the request.
Cheers,
M.

[1] https://docs.basex.org/wiki/Request_Module


On 22/04/20 12:18, Zimmel, Daniel wrote:

Hi Daniel,

RTFM. Simply pass the body in %rest:POST("{$body}").

Sorry for disturbing!

Daniel

-Ursprüngliche Nachricht-
Von: Zimmel, Daniel 
Gesendet: Mittwoch, 22. April 2020 11:27
An: BaseX 
Betreff: [basex-talk] A question about RESTXQ rest:form-param()

Hi,

in an HTML form with a RESTXQ POST action, do I really have to declare 
%rest:form-param() for every param, or is there a way to pass the full query 
component in one parameter?

Best, Daniel





Re: [basex-talk] A question about RESTXQ rest:form-param()

2020-04-22 Thread Marco Lettere

Hi Daniel,
another way is to use request:* functions [1] available for programmatic 
inspection of the request.

Cheers,
M.

[1] https://docs.basex.org/wiki/Request_Module


On 22/04/20 12:18, Zimmel, Daniel wrote:

Hi Daniel,

RTFM. Simply pass the body in %rest:POST("{$body}").

Sorry for disturbing!

Daniel

-Ursprüngliche Nachricht-
Von: Zimmel, Daniel 
Gesendet: Mittwoch, 22. April 2020 11:27
An: BaseX 
Betreff: [basex-talk] A question about RESTXQ rest:form-param()

Hi,

in an HTML form with a RESTXQ POST action, do I really have to declare 
%rest:form-param() for every param, or is there a way to pass the full query 
component in one parameter?

Best, Daniel





Re: [basex-talk] New Slack workspace for XML discussions: xmlcom.slack.com

2020-04-17 Thread Marco Lettere

Yeah! I'm in it!
M.

On 17/04/20 15:35, Christian Grün wrote:

Dear XML aficionados,

For interactive discussions on XML, XQuery and BaseX, feel free to
join the new XML workspace on Slack:

https://join.slack.com/t/xmlcom/shared_invite/zt-dkmwrygj-kdviJUKoXb7_ERdCoJSAfQ

Thanks to Adam Retter for opening the workspace.
Thanks to Steven Majewski for forwarding the link.

Have fun,
Christian
__ _  _   _ _

On Thu, Apr 16, 2020 at 7:34 PM Majewski, Steven Dennis (sdm7g)
 wrote:

FYI:

Begin forwarded message:

From: "Adam Retter adam.ret...@googlemail.com" 

Subject: [xsl] [OT] New Slack workspace for XML discussions
Date: April 16, 2020 at 11:59:25 AM EDT
To: xsl-l...@lists.mulberrytech.com
Reply-To: 

Hopefully the moderators won't mind me posting this? I think it has
value for readers of this list...

Just a quick heads up in-case anyone is interested, there is now a new
vendor neutral Slack Workspace for discussing all things XML related -
https://www.xml.com/news/2020-04-slack-workspace-for-the-xml-community/

Cheers Adam.
--
Adam Retter

skype: adam.retter
tweet: adamretter
http://www.adamretter.org.uk
--~
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
EasyUnsubscribe: http://lists.mulberrytech.com/unsub/xsl-list/504751
or by email: xsl-list-un...@lists.mulberrytech.com
--~--






Re: [basex-talk] BaseX 9.3.2: Performance Tweaks, Bug Fixes

2020-03-10 Thread Marco Lettere
Currently we are not very used to good news ... So, thanks Christian for 
this one who definitely sounds like that!


M.

On 10/03/20 14:18, Christian Grün wrote:

Dear all,

We are glad to provide you with a new BaseX 9.3.2 maintenance release:

http://basex.org

In preparation to version 9.4, we have introduced numerous small query
optimizations that speed up path, predicate and simple map operations
and will rewrite your queries for index access more often than before.
Feel free to check out the full list of added rewritings and
enhancements on Github:

   https://github.com/BaseXdb/basex/issues?q=is%3Aissue+is%3Aclosed

Have fun with the new release,

Christian
BaseX Team





Re: [basex-talk] BaseX 9.3: The Winter Edition

2019-11-29 Thread Marco Lettere

Thank you all so much!
The list of improvements is really impressive!
Great job as usual.
M.

On 29/11/19 17:14, Christian Grün wrote:

Hello to our loyal readers of our list,

We are glad to announce version 9.3 of BaseX, our XML framework,
database system and XQuery 3.1 processor:

   http://basex.org/

The focus has been put on in-depth optimizations of complex XQuery
code and faster processing of RESTXQ web applications:

RESTXQ
- seamless streaming of custom responses to the client
- Request Module: bind attributes to current request
- better support for the OPTIONS and HEAD methods
- trace output is now stored in the database logs
- enforce premature HTTP response with web:error
- optional GZIP compression of HTTP responses
- server-side forwarding with web:forward

USER INTERFACES
- GUI: project view: skip directories with .ignore file
- GUI: project view: do now follow symbolic links
- DBA: user panel updated to show user info
- DBA: faster processing of log entries

STORAGE
- hot fix for replacing single documents with namespaces
- improved XQuery processing of binary resources
- ADDCACHE: faster caching of single documents
- WITHDB: less locking when using fn:doc

XQUERY FUNCTIONS
- user:info/user:update-info: process user-specific data
- db:open-id, db:open-pre: support for multiple ids
- file:descendants: return descendant file paths
- inspect:type: return type of a value as string
- db:alter-backup: rename database backups
- xslt:transform: support for catalog files

XQUERY PERFORMANCE
- merge of adjacent/redundant steps in paths and predicates
- removal of redundant switch, typeswitch and catch clauses
- interruption of side-effecting code (Files Modul etc.)
- fn:count: pre-evaluation of zero-or-one arguments
- faster processing of alternative steps: a/(b,c)
- rewriting of paths to simple map expressions
- inlining of values in simple map expressions
- flattening of nested simple map expressions
- rewriting of lists to union expressions
- removal of superfluous predicates

For a more comprehensive list of added and updated features. please
check out our documentation: http://docs.basex.org/.

Your feedback is welcome as usual.

Have fun with the new release,
Your BaseX Team





Re: [basex-talk] file:list and symbolic links

2019-11-09 Thread Marco Lettere
I see your point Christian. Thank you.
Of course recursively following the dir structure may work.
Passing a boolean to switch the behaviour with a developer choice might be
viable? Or is the file:list function standardized by xquery or expath spec?
Have a nice weekend!
M.

Il sab 9 nov 2019, 18:36 Christian Grün  ha
scritto:

> Hi Marco,
>
> The rationale is that symbolic links can lead to cycles if you have
> circular dependencies in your directory structure. What I learnt is
> that this happens quite frequently in practice.
>
> You can write an XQuery function with file:list and file:children,
> which recursively calls itself for each directory entry. The following
> function returns paths to all descendant files, excluding directories;
> it should follow all symbolic links:
>
>   declare function file:traverse($dir) {
> for $child in file:children($dir)
> return if (file:is-dir($child)) then (
>   file:traverse($child)
> ) else (
>   $child
> )
>   };
>
> Hope this helps,
> Christian
>
>
> On Wed, Nov 6, 2019 at 11:41 AM Marco Lettere  wrote:
> >
> > Dear all,
> >
> > what is the rationale behind the fact that file:list has been rewritten
> > to not following symbolic links [1]?
> >
> > We had some jobs related code that relied on symlinks and stopped
> > working after moving to new version of BaseX.
> >
> > Is there any work around? By using children or the new descendants
> > functions maybe?
> >
> > Thanks for your support.
> >
> > Marco.
> >
> > [1]
> >
> https://github.com/BaseXdb/basex/blob/bbb62d1ec3e10b4fa1cc48bd4e16911fa73e46df/basex-core/src/main/java/org/basex/query/func/file/FileList.java#L103
> >
>


[basex-talk] file:list and symbolic links

2019-11-06 Thread Marco Lettere

Dear all,

what is the rationale behind the fact that file:list has been rewritten 
to not following symbolic links [1]?


We had some jobs related code that relied on symlinks and stopped 
working after moving to new version of BaseX.


Is there any work around? By using children or the new descendants 
functions maybe?


Thanks for your support.

Marco.

[1] 
https://github.com/BaseXdb/basex/blob/bbb62d1ec3e10b4fa1cc48bd4e16911fa73e46df/basex-core/src/main/java/org/basex/query/func/file/FileList.java#L103




Re: [basex-talk] Database corrupted on updating with RestXQ

2019-09-26 Thread Marco Lettere

Yes!
I can confirm that the bug disappeared on the scenario used to reproduce it!
Thank again.
Marco.

On 26/09/19 14:37, Christian Grün wrote:


Is it automatically available shipped in docker?


I guess so ("Updated 38 minutes ago"):

https://hub.docker.com/u/basex



The current scenario where to put it immediately at test is based
on the
docker distribution and it's taking too much time at the moment to
setup
a different test env.
I've received your answer about concurrent XQuery execution. I'm
currently elaborating my observations on this. Hop to be able to
formulate them during WE.
Cheers,
Marco.

On 26/09/19 13:55, Christian Grün wrote:
> Hi Marco,
>
> Finally some news on the storage bug you reported: It is fixed with
> the latest snapshot [1].
>
> Looking forward to your tests (and I hope you received my
response on
> parallel querying),
> Christian
>
> [1] http://files.basex.org/releases/latest/
>
>
>
> On Tue, Aug 6, 2019 at 9:38 AM Christian Grün
mailto:christian.gr...@gmail.com>> wrote:
>> Hi Marco,
>>
>> I think that the bug fix (which is still on my todo list) will
be made
>> available with BaseX 9.3; so, for now, it’s probably better to
choose
>> the workaround.
>>
    >> Cheers,
>> Christian
>>
>>
>>
>> On Sat, Aug 3, 2019 at 10:02 AM Marco Lettere
mailto:m.lett...@gmail.com>> wrote:
>>> Hi Christian,
>>> I'm currently preparing a deployment based on Docker for one
of our
>>> customers. Here in Italy it's holiday time in August so I have
a bit of
>>> time and I can coope with your suggested workaround for the
moment.
>>> Just a question to be prepared ... are intermediate bug-fix
releases
>>> also available in the form of docker containers?
>>> Thanks for your support as usual.
>>> Regards,
>>> Marco.
>>>
>>> On 02/08/19 15:04, Christian Grün wrote:
>>>> Hi Marco,
>>>>
>>>> A little update on the status quo:
>>>>
>>>> • I noticed that the bug you observed was basically exposed
by the
>>>> previous bug fix (it existed before, but it didn’t occur in your
>>>> particular setting).
>>>> • It happens only in certain cases (i.e., for specific
document sizes)
>>>> and if the added/replaced document has namespaces.
>>>> • Another good thing: It happens only if your database is empty.
>>>>
>>>> It’s not exactly an elegant proposal, but as long as we
haven’t fixed
>>>> the bug, you could add an initial  document to your
database.
>>>>
>>>> We are doing our best, though.
>>>> Christian
>>>>
>>>>
>>>>
>>>> On Wed, Jul 31, 2019 at 10:20 AM Christian Grün
>>>> mailto:christian.gr...@gmail.com>> wrote:
>>>>> Marco, thanks for the attachment. I have created a script that
>>>>> triggers the error [1]. Most probably, the bug is related to the
>>>>> previous bug issue. We’ll have a look at this soon. – Christian
>>>>>
>>>>> [1] https://github.com/BaseXdb/basex/issues/1711
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Jul 27, 2019 at 4:02 PM Marco Lettere
mailto:m.lett...@gmail.com>> wrote:
>>>>>> Thanks Christian,
>>>>>> I haven't been able to reproduce the bug with your SSCE.
>>>>>> Nevertheless I spent some time in tracing the operations
with the actual
>>>>>> data I'm currently storing into the DB and a sequence of
>>>>>> db:add/db:replace close enough to actual app flow.
>>>>>> I attach the query script which should be rather self
explaining.
>>>>>> You may run the script changing always $f with $f + 1
starting (from 0).
>>>>>> By uncommenting the different configurations of the
variable $input you
>>>>>> can see how the misbehaviour depends somehow on the data
because only
>>>>>> $d8 is causing the crash.
>>>>>> I stared at the diffs from $d8 to the other but there
really isn't any
>>>>>> significant difference so now it's very hard

Re: [basex-talk] Database corrupted on updating with RestXQ

2019-09-26 Thread Marco Lettere

Thanks Christian.
Is it automatically available shipped in docker?
The current scenario where to put it immediately at test is based on the 
docker distribution and it's taking too much time at the moment to setup 
a different test env.
I've received your answer about concurrent XQuery execution. I'm 
currently elaborating my observations on this. Hop to be able to 
formulate them during WE.

Cheers,
Marco.

On 26/09/19 13:55, Christian Grün wrote:

Hi Marco,

Finally some news on the storage bug you reported: It is fixed with
the latest snapshot [1].

Looking forward to your tests (and I hope you received my response on
parallel querying),
Christian

[1] http://files.basex.org/releases/latest/



On Tue, Aug 6, 2019 at 9:38 AM Christian Grün  wrote:

Hi Marco,

I think that the bug fix (which is still on my todo list) will be made
available with BaseX 9.3; so, for now, it’s probably better to choose
the workaround.

Cheers,
Christian



On Sat, Aug 3, 2019 at 10:02 AM Marco Lettere  wrote:

Hi Christian,
I'm currently preparing a deployment based on Docker for one of our
customers. Here in Italy it's holiday time in August so I have a bit of
time and I can coope with your suggested workaround for the moment.
Just a question to be prepared ... are intermediate bug-fix releases
also available in the form of docker containers?
Thanks for your support as usual.
Regards,
Marco.

On 02/08/19 15:04, Christian Grün wrote:

Hi Marco,

A little update on the status quo:

• I noticed that the bug you observed was basically exposed by the
previous bug fix (it existed before, but it didn’t occur in your
particular setting).
• It happens only in certain cases (i.e., for specific document sizes)
and if the added/replaced document has namespaces.
• Another good thing: It happens only if your database is empty.

It’s not exactly an elegant proposal, but as long as we haven’t fixed
the bug, you could add an initial  document to your database.

We are doing our best, though.
Christian



On Wed, Jul 31, 2019 at 10:20 AM Christian Grün
 wrote:

Marco, thanks for the attachment. I have created a script that
triggers the error [1]. Most probably, the bug is related to the
previous bug issue. We’ll have a look at this soon. – Christian

[1] https://github.com/BaseXdb/basex/issues/1711



On Sat, Jul 27, 2019 at 4:02 PM Marco Lettere  wrote:

Thanks Christian,
I haven't been able to reproduce the bug with your SSCE.
Nevertheless I spent some time in tracing the operations with the actual
data I'm currently storing into the DB and a sequence of
db:add/db:replace close enough to actual app flow.
I attach the query script which should be rather self explaining.
You may run the script changing always $f with $f + 1 starting (from 0).
By uncommenting the different configurations of the variable $input you
can see how the misbehaviour depends somehow on the data because only
$d8 is causing the crash.
I stared at the diffs from $d8 to the other but there really isn't any
significant difference so now it's very hard for me to understand.

Thank you in advance for all support that you can provide as usual.
Regards,
Marco.

On 26/07/19 18:41, Christian Grün wrote:

Hi Marco,

Back in February, a user reported a database inconsistency [1] – which
happened, too, if a database was nearly empty – and we managed to
build a little text case to get this reproduced [2]. After that, 9.2
was released. Maybe this fix introduced another irregularity for this
special case?

Maybe we can get this reproduced by taking the script from issue 1662
and modify it a little? That would be great.

Sorry to the inconvenience,
Christian

PS: You can safely ignore the "Creating database..." output. It may
just indicate that an XML document is parsed, and that an internal
main-memory database instance is created, possibly for your users.xml
file in the data directory.

[1] 
https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg11459.html
[2] https://github.com/BaseXdb/basex/issues/1662



On Fri, Jul 26, 2019 at 4:29 PM Marco Lettere  wrote:

Hi all,

starting with 9.2.1 we experience a strange error with a RESTXQ API  of
ours that we have been using for years.

The typical pattern is lookup a document update it and store it back.

We have done this milions of time and also all the tests work neatly.

But when using it from inside the web application, at around the tenth
operation we get the DB corrupted with the stack trace [1]. This seems
to happen only when the database is nearly empty and, for sure, it
doesn't appear on 8.x series.

It feels like some concurrency flaw is introduced somewhere but it's
hard to say because the API is eight years in age and it never changed
significantly.

I know it's hard without an SSCE... but while working to insulate this
byzantine behaviour (I've been trying for days now) maybe someone in the
list has a clue or is able to interpret the stack trace?

Thanks and have a nice weekend,

Marco.


[1]

Im

[basex-talk] Suggestion for concurrency of RestXQ

2019-09-17 Thread Marco Lettere

Dear all,

on one of our applications based on RestXQ services running on 9.0.2 we 
experience a strange behaviour related to concurrency.


It is a stateless service running on a server with http-local set to 
true. So no database access. We removed all accesses to sessions and the 
code doesn't even write to files or other external resources. It's 
really only computation.


When I run multiple requests, I can clearly see that they are accepted 
simultaneously (up to 8 which matches PARALLEL). They even complete out 
of order according to the session id reported in the log. [1]


The strange thing is: performance seems to be as if calls were executed 
serially. If T is the time to complete one call then the pattern is T + 
T + T + T + T.


I have to add that the algorithm is rather complex. Lots of functions 
(possibly inlined, private and the main one even recursive) which filter 
and transform a large XML request into a possibly large XML output.


If I simplify the called RestXQ to something like [2] then I get a log 
like [3] which shows a pattern like the one I'm expecting T+d+d+d+d 
where d is just a slight overhead (orders of magnitude smaller).


My question is what do I have to check in order to understand where the 
hard synchronization of the calls occurs? Might this be related to 
non-deterministic functions? What else?


I've tried to isolate parts of the code but I'm unable to get 
predictable results.


Thank you very much for all the hints you will be able to provide!

Regards,

Marco.

[1]

17:10:19.204    127.0.0.1:58038    admin    REQUEST    [POST] 
http://localhost:8081/rate-selector/select-matching-rates
17:10:19.204    127.0.0.1:58040    admin    REQUEST [POST] 
http://localhost:8081/rate-selector/select-matching-rates
17:10:19.205    127.0.0.1:58042    admin    REQUEST [POST] 
http://localhost:8081/rate-selector/select-matching-rates
17:10:19.205    127.0.0.1:58044    admin    REQUEST [POST] 
http://localhost:8081/rate-selector/select-matching-rates
17:10:19.208    127.0.0.1:58046    admin    REQUEST [POST] 
http://localhost:8081/rate-selector/select-matching-rates

17:10:21.325    127.0.0.1:58040    admin    200 2121.01 ms
17:10:23.174    127.0.0.1:58046    admin    200 3965.74 ms
17:10:25.035    127.0.0.1:58038    admin    200 5830.63 ms
17:10:26.889    127.0.0.1:58044    admin    200 7683.76 ms
17:10:28.739    127.0.0.1:58042    admin    200 9534.46 ms


[2]

(:~
 : Test concurrency
 :)
declare
  %rest:path("/test")
  %rest:POST("{$body}")
function rs:test($body as node()) {
   {$body}{fold-left(1 to 1000, 0, function($acc, $next){ $acc + 
math:sqrt($next)})}

};


[3]

17:48:02.782    127.0.0.1:48262    admin    REQUEST    [POST] 
http://localhost:8081/rate-selector/test
17:48:02.783    127.0.0.1:48252    admin    REQUEST    [POST] 
http://localhost:8081/rate-selector/test
17:48:02.783    127.0.0.1:48258    admin    REQUEST    [POST] 
http://localhost:8081/rate-selector/test
17:48:02.784    127.0.0.1:48260    admin    REQUEST    [POST] 
http://localhost:8081/rate-selector/test
17:48:02.784    127.0.0.1:48254    admin    REQUEST    [POST] 
http://localhost:8081/rate-selector/test

17:48:04.428    127.0.0.1:48262    admin    200        1646.03 ms
17:48:04.509    127.0.0.1:48260    admin    200        1725.45 ms
17:48:04.710    127.0.0.1:48254    admin    200        1926.27 ms
17:48:04.772    127.0.0.1:48258    admin    200        1988.41 ms
17:48:04.904    127.0.0.1:48252    admin    200        2120.33 ms




Re: [basex-talk] Database corrupted on updating with RestXQ

2019-08-03 Thread Marco Lettere

Hi Christian,
I'm currently preparing a deployment based on Docker for one of our 
customers. Here in Italy it's holiday time in August so I have a bit of 
time and I can coope with your suggested workaround for the moment.
Just a question to be prepared ... are intermediate bug-fix releases 
also available in the form of docker containers?

Thanks for your support as usual.
Regards,
Marco.

On 02/08/19 15:04, Christian Grün wrote:

Hi Marco,

A little update on the status quo:

• I noticed that the bug you observed was basically exposed by the
previous bug fix (it existed before, but it didn’t occur in your
particular setting).
• It happens only in certain cases (i.e., for specific document sizes)
and if the added/replaced document has namespaces.
• Another good thing: It happens only if your database is empty.

It’s not exactly an elegant proposal, but as long as we haven’t fixed
the bug, you could add an initial  document to your database.

We are doing our best, though.
Christian



On Wed, Jul 31, 2019 at 10:20 AM Christian Grün
 wrote:

Marco, thanks for the attachment. I have created a script that
triggers the error [1]. Most probably, the bug is related to the
previous bug issue. We’ll have a look at this soon. – Christian

[1] https://github.com/BaseXdb/basex/issues/1711



On Sat, Jul 27, 2019 at 4:02 PM Marco Lettere  wrote:

Thanks Christian,
I haven't been able to reproduce the bug with your SSCE.
Nevertheless I spent some time in tracing the operations with the actual
data I'm currently storing into the DB and a sequence of
db:add/db:replace close enough to actual app flow.
I attach the query script which should be rather self explaining.
You may run the script changing always $f with $f + 1 starting (from 0).
By uncommenting the different configurations of the variable $input you
can see how the misbehaviour depends somehow on the data because only
$d8 is causing the crash.
I stared at the diffs from $d8 to the other but there really isn't any
significant difference so now it's very hard for me to understand.

Thank you in advance for all support that you can provide as usual.
Regards,
Marco.

On 26/07/19 18:41, Christian Grün wrote:

Hi Marco,

Back in February, a user reported a database inconsistency [1] – which
happened, too, if a database was nearly empty – and we managed to
build a little text case to get this reproduced [2]. After that, 9.2
was released. Maybe this fix introduced another irregularity for this
special case?

Maybe we can get this reproduced by taking the script from issue 1662
and modify it a little? That would be great.

Sorry to the inconvenience,
Christian

PS: You can safely ignore the "Creating database..." output. It may
just indicate that an XML document is parsed, and that an internal
main-memory database instance is created, possibly for your users.xml
file in the data directory.

[1] 
https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg11459.html
[2] https://github.com/BaseXdb/basex/issues/1662



On Fri, Jul 26, 2019 at 4:29 PM Marco Lettere  wrote:

Hi all,

starting with 9.2.1 we experience a strange error with a RESTXQ API  of
ours that we have been using for years.

The typical pattern is lookup a document update it and store it back.

We have done this milions of time and also all the tests work neatly.

But when using it from inside the web application, at around the tenth
operation we get the DB corrupted with the stack trace [1]. This seems
to happen only when the database is nearly empty and, for sure, it
doesn't appear on 8.x series.

It feels like some concurrency flaw is introduced somewhere but it's
hard to say because the API is eight years in age and it never changed
significantly.

I know it's hard without an SSCE... but while working to insulate this
byzantine behaviour (I've been trying for days now) maybe someone in the
list has a clue or is able to interpret the stack trace?

Thanks and have a nice weekend,

Marco.


[1]

Improper use? Potential bug? Your feedback is welcome:
Contact: basex-talk@mailman.uni-konstanz.de
Version: BaseX 9.3 beta
Java: Ubuntu, 13
OS: Linux, amd64
Stack Trace:
java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
   at org.basex.io.random.TableDiskAccess.fpre(TableDiskAccess.java:507)
   at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468)
   at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:156)
   at org.basex.data.Data.kind(Data.java:304)
   at org.basex.query.value.node.DBNode.(DBNode.java:50)
   at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:139)
   at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:164)
   at org.basex.query.func.db.DbOpen.value(DbOpen.java:19)
   at org.basex.query.func.StandardFunc.optimize(StandardFunc.java:82)
   at org.basex.query.expr.Arr.inline(Arr.java:69)
   at org.basex.query.expr.path.Path.inline(Pat

Re: [basex-talk] Forcing download of JSON doc

2019-08-01 Thread Marco Lettere

Maybe using json:serialize() rather than serialize?

In general if output:method is set to Json it should be sufficient to 
return map{} or array{} or alternatively the XML version of a json 
document without having to explicitly serialize ...


M.

On 01/08/19 14:36, Tim Thompson wrote:
Thank you, Marco. So, the download issue seems to be with XForms 
(XSLTForms) rather than BaseX. If I do a file upload from a plain HTML 
form, the file download is triggered correctly. However, even though I 
have set "application/json" as the Content-Type, the contents of the 
downloaded file are an escaped string rather than a serialized JSON 
object: e.g.,


"{\n  \"key\": \"value\"\n}"
Is there a way to produce a properly serialized JSON file for download?

--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library

On Thu, Aug 1, 2019 at 3:03 AM Marco Lettere <mailto:m.lett...@gmail.com>> wrote:


Hi Tim,
downloading from a browser when posting a form is usually a bit
tricky. For me it always ended up with the creation of an
artificial  element with an href crafted from the form
parameters and with an attribute download="filename.ext" added to it.
Anyway you could try with returning a "forced"
application/octet-stream Content-Type header 

M.

On 01/08/19 05:00, Tim Thompson wrote:

Hello,

I am posting a JSON doc to a RESTXQ function from XForms and
trying to set a Content-Disposition header to trigger a file
download. Is this possible to do without first saving the JSON
doc as a file? Right now, the RESTXQ function returns the correct
headers, but does not force a file download dialog.

Thanks in advance!
Tim

--
Query:

xquery version "3.1";

module namespace wrap = "http://example.org/wrap;;

declare
  %rest:path("/wrap")
  %rest:POST("{$data}")
  %output:method("json")
function wrap:process(
  $data as node()
) as item()* {
  let $name := data($data/data/upload/file/@name)
  let $json := json-to-xml($data/data/upload/file)
  let $header :=
    ``[attachment; filename="`{$name}`";]``
  return
    (
    
      
        
        
      
    ,

    http://www.w3.org/2005/xpath-functions;>
        
          {$json}
          Wrapper_Profile
          Wrapper:Profile:BF2
          Profile wrapper for resource
templates
          Wrapper
          https://ld4p.github.io/sinopia/schemas/0.0.9/profile.json
          {current-dateTime()}
        
     => xml-to-json() => parse-json() => serialize(map
{'method': 'json', 'use-character-maps' : map {'/' : '/'}})
    )
};


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library







Re: [basex-talk] Forcing download of JSON doc

2019-08-01 Thread Marco Lettere

Hi Tim,
downloading from a browser when posting a form is usually a bit tricky. 
For me it always ended up with the creation of an artificial  
element with an href crafted from the form parameters and with an 
attribute download="filename.ext" added to it.
Anyway you could try with returning a "forced" application/octet-stream 
Content-Type header 


M.

On 01/08/19 05:00, Tim Thompson wrote:

Hello,

I am posting a JSON doc to a RESTXQ function from XForms and trying to 
set a Content-Disposition header to trigger a file download. Is this 
possible to do without first saving the JSON doc as a file? Right now, 
the RESTXQ function returns the correct headers, but does not force a 
file download dialog.


Thanks in advance!
Tim

--
Query:

xquery version "3.1";

module namespace wrap = "http://example.org/wrap;;

declare
  %rest:path("/wrap")
  %rest:POST("{$data}")
  %output:method("json")
function wrap:process(
  $data as node()
) as item()* {
  let $name := data($data/data/upload/file/@name)
  let $json := json-to-xml($data/data/upload/file)
  let $header :=
    ``[attachment; filename="`{$name}`";]``
  return
    (
    
      
        
        
      
    ,

    http://www.w3.org/2005/xpath-functions;>
        
          {$json}
          Wrapper_Profile
          Wrapper:Profile:BF2
                      key="description">Profile wrapper for resource 
templates

          Wrapper
                      
key="schema">https://ld4p.github.io/sinopia/schemas/0.0.9/profile.json

          {current-dateTime()}
        
     => xml-to-json() => parse-json() => serialize(map 
{'method': 'json', 'use-character-maps' : map {'/' : '/'}})

    )
};


--
Tim A. Thompson
Discovery Metadata Librarian
Yale University Library





Re: [basex-talk] Database corrupted on updating with RestXQ

2019-07-27 Thread Marco Lettere

Thanks Christian,
I haven't been able to reproduce the bug with your SSCE.
Nevertheless I spent some time in tracing the operations with the actual 
data I'm currently storing into the DB and a sequence of 
db:add/db:replace close enough to actual app flow.

I attach the query script which should be rather self explaining.
You may run the script changing always $f with $f + 1 starting (from 0).
By uncommenting the different configurations of the variable $input you 
can see how the misbehaviour depends somehow on the data because only 
$d8 is causing the crash.
I stared at the diffs from $d8 to the other but there really isn't any 
significant difference so now it's very hard for me to understand.


Thank you in advance for all support that you can provide as usual.
Regards,
Marco.

On 26/07/19 18:41, Christian Grün wrote:

Hi Marco,

Back in February, a user reported a database inconsistency [1] – which
happened, too, if a database was nearly empty – and we managed to
build a little text case to get this reproduced [2]. After that, 9.2
was released. Maybe this fix introduced another irregularity for this
special case?

Maybe we can get this reproduced by taking the script from issue 1662
and modify it a little? That would be great.

Sorry to the inconvenience,
Christian

PS: You can safely ignore the "Creating database..." output. It may
just indicate that an XML document is parsed, and that an internal
main-memory database instance is created, possibly for your users.xml
file in the data directory.

[1] 
https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg11459.html
[2] https://github.com/BaseXdb/basex/issues/1662



On Fri, Jul 26, 2019 at 4:29 PM Marco Lettere  wrote:

Hi all,

starting with 9.2.1 we experience a strange error with a RESTXQ API  of
ours that we have been using for years.

The typical pattern is lookup a document update it and store it back.

We have done this milions of time and also all the tests work neatly.

But when using it from inside the web application, at around the tenth
operation we get the DB corrupted with the stack trace [1]. This seems
to happen only when the database is nearly empty and, for sure, it
doesn't appear on 8.x series.

It feels like some concurrency flaw is introduced somewhere but it's
hard to say because the API is eight years in age and it never changed
significantly.

I know it's hard without an SSCE... but while working to insulate this
byzantine behaviour (I've been trying for days now) maybe someone in the
list has a clue or is able to interpret the stack trace?

Thanks and have a nice weekend,

Marco.


[1]

Improper use? Potential bug? Your feedback is welcome:
Contact: basex-talk@mailman.uni-konstanz.de
Version: BaseX 9.3 beta
Java: Ubuntu, 13
OS: Linux, amd64
Stack Trace:
java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
  at org.basex.io.random.TableDiskAccess.fpre(TableDiskAccess.java:507)
  at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468)
  at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:156)
  at org.basex.data.Data.kind(Data.java:304)
  at org.basex.query.value.node.DBNode.(DBNode.java:50)
  at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:139)
  at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:164)
  at org.basex.query.func.db.DbOpen.value(DbOpen.java:19)
  at org.basex.query.func.StandardFunc.optimize(StandardFunc.java:82)
  at org.basex.query.expr.Arr.inline(Arr.java:69)
  at org.basex.query.expr.path.Path.inline(Path.java:962)
  at org.basex.query.expr.gflwor.ForLet.inline(ForLet.java:66)
  at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:804)
  at org.basex.query.expr.gflwor.GFLWOR.inlineLets(GFLWOR.java:339)
  at org.basex.query.expr.gflwor.GFLWOR.optimize(GFLWOR.java:106)
  at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:785)
  at org.basex.query.expr.Try.inline(Try.java:106)
  at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:816)
  at org.basex.query.expr.gflwor.GFLWOR.inlineLets(GFLWOR.java:339)
  at org.basex.query.expr.gflwor.GFLWOR.optimize(GFLWOR.java:106)
  at org.basex.query.func.StaticFunc.inline(StaticFunc.java:294)
  at org.basex.query.func.StaticFuncCall.compile(StaticFuncCall.java:67)
  at org.basex.query.scope.MainModule.comp(MainModule.java:81)
  at org.basex.query.QueryCompiler.compile(QueryCompiler.java:114)
  at org.basex.query.QueryCompiler.compile(QueryCompiler.java:105)
  at org.basex.query.QueryContext.compile(QueryContext.java:312)
  at org.basex.query.QueryContext.iter(QueryContext.java:331)
  at
org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:73)
  at org.basex.http.web.WebResponse.create(WebResponse.java:63)
  at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:53)
  at org.basex.http.BaseXServlet.service(BaseXServl

Re: [basex-talk] Database corrupted on updating with RestXQ

2019-07-26 Thread Marco Lettere

As an additional remark I set the MIXUPDATES = true option.
Moreover I notice the wording "Creating database..." among the request 
logs that are output when running with -d [1].
This sounds strange (the only point I see the string in the code is the 
Builder class [2]) but it happens also with 8.6.7 which doesn't show 
this misbehaviour.


M.

[1]
_ REQUEST _
(GET /hlcm/xdwapi/v2/xdw/2.25.8046764881955849894)@912634350 
org.eclipse.jetty.server.Request@3665b1ee

- Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
- Connection: keep-alive
- User-Agent: Java/13
- Host: 10.17.217.1:8984
_ RESPONSE 
HTTP/1.1 200
Set-Cookie: JSESSIONID=a2ryzgdm4t941ra9jmes9mj87;Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/xml; charset=UTF-8
*
Creating Database...*
. 5.58 ms (100 MB)
_ REQUEST _
(GET 
/hlcm/taskviewer/showworkflow?workflowid=2.25.8046764881955849894)@1286045891 
org.eclipse.jetty.server.Request@4ca780c3

- Accept: */*
- Connection: keep-alive
- Referer: 
http://localhost:8984/ui/hlcm/main.cameo?config=HLCM_1=test1=rossi

- Accept-Encoding: gzip, deflate, br
- User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, 
like Gecko) Ubuntu Chromium/75.0.3770.90 Chrome/75.0.3770.90 Safari/537.36

- Accept-Language: it,en-US;q=0.9,en;q=0.8
- Cookie: COOKIE_SUPPORT=true; GUEST_LANGUAGE_ID=en_US; pnctest=1; 
JSESSIONID=2g5ep7mzq0ah1ktre4b398hfc

- Host: localhost:8984
_ RESPONSE 
HTTP/1.1 200

[2] 
https://github.com/BaseXdb/basex/blob/f7a5492c46d55e1c1f58df24b8ed9567c176e8c1/basex-core/src/main/java/org/basex/build/Builder.java


On 26/07/19 16:29, Marco Lettere wrote:

Hi all,

starting with 9.2.1 we experience a strange error with a RESTXQ API  
of ours that we have been using for years.


The typical pattern is lookup a document update it and store it back.

We have done this milions of time and also all the tests work neatly.

But when using it from inside the web application, at around the tenth 
operation we get the DB corrupted with the stack trace [1]. This seems 
to happen only when the database is nearly empty and, for sure, it 
doesn't appear on 8.x series.


It feels like some concurrency flaw is introduced somewhere but it's 
hard to say because the API is eight years in age and it never changed 
significantly.


I know it's hard without an SSCE... but while working to insulate this 
byzantine behaviour (I've been trying for days now) maybe someone in 
the list has a clue or is able to interpret the stack trace?


Thanks and have a nice weekend,

Marco.


[1]

Improper use? Potential bug? Your feedback is welcome:
Contact: basex-talk@mailman.uni-konstanz.de
Version: BaseX 9.3 beta
Java: Ubuntu, 13
OS: Linux, amd64
Stack Trace:
java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for 
length 1

    at org.basex.io.random.TableDiskAccess.fpre(TableDiskAccess.java:507)
    at 
org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468)
    at 
org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:156)

    at org.basex.data.Data.kind(Data.java:304)
    at org.basex.query.value.node.DBNode.(DBNode.java:50)
    at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:139)
    at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:164)
    at org.basex.query.func.db.DbOpen.value(DbOpen.java:19)
    at org.basex.query.func.StandardFunc.optimize(StandardFunc.java:82)
    at org.basex.query.expr.Arr.inline(Arr.java:69)
    at org.basex.query.expr.path.Path.inline(Path.java:962)
    at org.basex.query.expr.gflwor.ForLet.inline(ForLet.java:66)
    at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:804)
    at org.basex.query.expr.gflwor.GFLWOR.inlineLets(GFLWOR.java:339)
    at org.basex.query.expr.gflwor.GFLWOR.optimize(GFLWOR.java:106)
    at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:785)
    at org.basex.query.expr.Try.inline(Try.java:106)
    at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:816)
    at org.basex.query.expr.gflwor.GFLWOR.inlineLets(GFLWOR.java:339)
    at org.basex.query.expr.gflwor.GFLWOR.optimize(GFLWOR.java:106)
    at org.basex.query.func.StaticFunc.inline(StaticFunc.java:294)
    at 
org.basex.query.func.StaticFuncCall.compile(StaticFuncCall.java:67)

    at org.basex.query.scope.MainModule.comp(MainModule.java:81)
    at org.basex.query.QueryCompiler.compile(QueryCompiler.java:114)
    at org.basex.query.QueryCompiler.compile(QueryCompiler.java:105)
    at org.basex.query.QueryContext.compile(QueryContext.java:312)
    at org.basex.query.QueryContext.iter(QueryContext.java:331)
    at 
org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:73)

    at org.basex.http.web.WebResponse.create(WebResponse.java:63)
    at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:53)
    at org.basex.http.BaseXServlet.service(BaseXServl

[basex-talk] Database corrupted on updating with RestXQ

2019-07-26 Thread Marco Lettere

Hi all,

starting with 9.2.1 we experience a strange error with a RESTXQ API  of 
ours that we have been using for years.


The typical pattern is lookup a document update it and store it back.

We have done this milions of time and also all the tests work neatly.

But when using it from inside the web application, at around the tenth 
operation we get the DB corrupted with the stack trace [1]. This seems 
to happen only when the database is nearly empty and, for sure, it 
doesn't appear on 8.x series.


It feels like some concurrency flaw is introduced somewhere but it's 
hard to say because the API is eight years in age and it never changed 
significantly.


I know it's hard without an SSCE... but while working to insulate this 
byzantine behaviour (I've been trying for days now) maybe someone in the 
list has a clue or is able to interpret the stack trace?


Thanks and have a nice weekend,

Marco.


[1]

Improper use? Potential bug? Your feedback is welcome:
Contact: basex-talk@mailman.uni-konstanz.de
Version: BaseX 9.3 beta
Java: Ubuntu, 13
OS: Linux, amd64
Stack Trace:
java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
    at org.basex.io.random.TableDiskAccess.fpre(TableDiskAccess.java:507)
    at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468)
    at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:156)
    at org.basex.data.Data.kind(Data.java:304)
    at org.basex.query.value.node.DBNode.(DBNode.java:50)
    at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:139)
    at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:164)
    at org.basex.query.func.db.DbOpen.value(DbOpen.java:19)
    at org.basex.query.func.StandardFunc.optimize(StandardFunc.java:82)
    at org.basex.query.expr.Arr.inline(Arr.java:69)
    at org.basex.query.expr.path.Path.inline(Path.java:962)
    at org.basex.query.expr.gflwor.ForLet.inline(ForLet.java:66)
    at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:804)
    at org.basex.query.expr.gflwor.GFLWOR.inlineLets(GFLWOR.java:339)
    at org.basex.query.expr.gflwor.GFLWOR.optimize(GFLWOR.java:106)
    at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:785)
    at org.basex.query.expr.Try.inline(Try.java:106)
    at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:816)
    at org.basex.query.expr.gflwor.GFLWOR.inlineLets(GFLWOR.java:339)
    at org.basex.query.expr.gflwor.GFLWOR.optimize(GFLWOR.java:106)
    at org.basex.query.func.StaticFunc.inline(StaticFunc.java:294)
    at org.basex.query.func.StaticFuncCall.compile(StaticFuncCall.java:67)
    at org.basex.query.scope.MainModule.comp(MainModule.java:81)
    at org.basex.query.QueryCompiler.compile(QueryCompiler.java:114)
    at org.basex.query.QueryCompiler.compile(QueryCompiler.java:105)
    at org.basex.query.QueryContext.compile(QueryContext.java:312)
    at org.basex.query.QueryContext.iter(QueryContext.java:331)
    at 
org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:73)

    at org.basex.http.web.WebResponse.create(WebResponse.java:63)
    at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:53)
    at org.basex.http.BaseXServlet.service(BaseXServlet.java:65)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
    at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
    at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

    at org.eclipse.jetty.server.Server.handle(Server.java:505)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
    at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
    at 

Re: [basex-talk] Quickest way to parse a DomSource

2019-07-22 Thread Marco Lettere

Thank you Martin.
It's worth experimenting

In the meanwhile (thanks to my colleague who had a closer look at the 
DOMSource API) I got to understand that there's a getNode() method 
returning the underlying DOM Node.

This allows for minimizing the module import statements.

Thus, fyi,  at the moment this is my work-around:

(: Import DOMSource API:)
declare namespace source = "javax.xml.transform.dom.DOMSource";

(:... code to get to the custom Document representation stored in a 
variable called $document... :)


(: get the DOMSource through custom API:)
let $source := bpmndom:getDomSource($document)

(:DOMSource API to get underlying Document Node :)
let $doc := source:getNode($source)

(:Now this Xpath expression works :)
return $doc//*:userTask

Cheers,
Marco.


On 22/07/19 12:53, Martin Honnen wrote:

Am 22.07.2019 um 11:32 schrieb Marco Lettere:



I have to access portions of the XML representation of the BPMN file
by binding the engine's API.

The closest that I can get to a standard XML representation is
currently a javax.xml.transform.dom.DOMSource.

What is the shortest way to transform (in XQuery) this DomSource to a
BaseX representation of the Document in order to query it?



The XQJ API seems to have a method

http://xqj.net/javadoc/javax/xml/xquery/XQDynamicContext.html#bindDocument(javax.xml.namespace.QName,%20javax.xml.transform.Source,%20javax.xml.xquery.XQItemType) 



 to allow passing in a java.xml.transform.Source so that seems at least
one way to deal with a DOMSource. But it happens on the Java side and is
tied to that particular XQJ API.







[basex-talk] Quickest way to parse a DomSource

2019-07-22 Thread Marco Lettere

Hi all,

I'm currently working on the usage of BaseX Xquery scripts inside a BPMN 
engine.


I have to access portions of the XML representation of the BPMN file by 
binding the engine's API.


The closest that I can get to a standard XML representation is currently 
a javax.xml.transform.dom.DOMSource.


What is the shortest way to transform (in XQuery) this DomSource to a 
BaseX representation of the Document in order to query it?


Thanks,

Marco.




Re: [basex-talk] PATCH verb in RestXQ implementation

2019-07-17 Thread Marco Lettere

Hi Daniel,
I'm pretty sure that PATCH is supported in recent versions of BaseX. 
Even if I don't remember exactly when it started since it's not traced 
in the Changelog of the documentation page for RestXQ.


From the docs at [1]:

The HTTP method annotations are equivalent to allHTTP request methods 
except TRACE and 
CONNECT. Zero or more methods may be used on a function; if none is 
specified, the function will be invoked for each method.


[1] http://docs.basex.org/wiki/RESTXQ

Ciao,
Marco.

On 16/07/19 19:22, Schopper, Daniel wrote:

Dear list,
I just came across the fact that BaseX does not recognize PATCH as a 
known HTTP verb (at least in my admittedly not bleading-edge version). 
Is this a feature or a bug?
And, in order to work around that: is there a way of passing payload 
to a custom %rest:method (without having an additional PUT or POST 
annotation on the same XQuery function)?

Thanks in advance!
Best
Daniel





[basex-talk] [ANN.] JSR223 wrapper

2019-07-05 Thread Marco Lettere
Ok then ... since I haven't got any feedback on this and because 
crawling the web myself hasn't produced any results either, I decided to 
invest a couple of weekends re-studying the the JSR223 (Java Scripting 
Framework) and implementing a wrapper for BaseX by myself.


For those who are not aware, JSR223 is a specification standardizing the 
embedding scripting languages into Java based applications. Was BSF once 
upon a time
As a use case at Nubisware, in particular, we wanted to continue using 
BaseX powered XQuery 3.1 scripts inside BPMN workflow engines for 
ScriptTasks and Event Handlers.


One plan that we have for the future is to create an equivalent engine 
that is able to execute scripts through REST on a remote BaseX server 
... but still reasoning whether this is backed by any real use case.


You can access the code and readme at [1] or download the current v0.1 
jar from [2].


Of course every 2 cents of feedback will be warmly appreciated.

Enjoy,

Marco.

[1] https://github.com/nubisware/basexjsr223
[2] 
https://github.com/nubisware/basexjsr223/releases/download/v0.1/BaseXJSR223.jar


On 11/06/19 13:00, Marco Lettere wrote:

Hi all,

is anyone on the list aware of a JSR223 compliant wrapper for BaseX in 
order to include XQuery scripting "seamlessly" into scriptable 
applications?


Thanks.

Marco.





[basex-talk] JS223 wrapper

2019-06-11 Thread Marco Lettere

Hi all,

is anyone on the list aware of a JS223 compliant wrapper for BaseX in 
order to include XQuery scripting "seamlessly" into scriptable applications?


Thanks.

Marco.



Re: [basex-talk] Update of arbitrarily selected elements?

2019-06-03 Thread Marco Lettere
Yes. In addition, if you don't have IDs in place and elements are all in 
the same container, you could maybe use their position. Or a combination 
of an XPAT selector for the contianer root and then a sequence of 
positions for the elements to be updated.

M.

On 03/06/19 09:46, Michael Seiferle wrote:

Hi Josh,

I think the most widely used approach would be addressing items via ID 
— just as you would in a traditional database approach — this might 
involve adding IDs to all elements.



(as in, not selectable with a query)
With XPath you can select arbitrary parts of your document, maybe 
using XPath expressions would be helpful?



Hope this helps a bit, feel free to ask for more specific information  :)

Best

Michael







Am 31.05.2019 um 20:14 schrieb Joshua Kordani 
>:


Greetings all.
I am curious about how basex supports the idea of modifying any given 
(or set of given) elements, say a user makes a query to select a 
bunch of elements for editing and wishes to add an attribute to an 
arbitrary subset (as in, not selectable with a query), how do I 
reference a particular element for later updating?

Sincerely,
Josh






Re: [basex-talk] Optimizer and JSON/XML response

2019-05-17 Thread Marco Lettere

Hi Christian,
I'll try to answer your questions inline ...

On 16/05/19 19:38, Christian Grün wrote:

Hi Marco,

We haven’t experienced such a behavior by ourselves, even in very
complex applications. Is the faulty behavior reproducible?

Only with the exact configuration of our current applciation...

Does it
make a difference if you change the value of the PARSERESTXQ option
[1]?
No, it doesn't. we tried higher values (wrt the default 3) and also 0 
and -1 ...

And is it already the http:send-request call that returns the
wrong result, or is it the actual RESTXQ function in which the
http:send-request call is contained?
The http:send-request returns the second element of the response in map 
format. We discovered it actually because a following XPath expression 
raised an error because it hit a map instead of a document as expected...

You could additionally output the
result of the function call to standard error (via fn:trace) or to a
file (via file:write) and compare the output in both of your setups.
We did all sort of tests and it's the http:send-request without doubts 
that already returns a map...


If there’s any chance to reproduce this (e.g. by creating hundreds or
thousands of dummy RESTXQ functions), that’d be fine.


I've tried to generate #FILES * #FUNCT  number of RESTXQS with 
increasing number of files and functions (up to OOM exceptions with 
default -Xms=2G) but I'm unable to reproduce... [A][B]


So I suspect that the problem might be in he file structure  we use 
softlinks to hook up restxq folders, a REPO full of implementation 
modules so may be the resolution of all this relations might be the issue?


Clueless at the moment ...

M.

[A] python script to generate extra RESTXQS

#!/bin/python3
FILES = 100
RESTXQS = 1000

qtempl = open("template.txt").read()

for f in range(FILES):
    outf = open("file" + str(f) + ".xqm", "w")
    qout = 'module namespace _ = "urn:dummy:' + str(f) + '";\n\n';
    for q in range(RESTXQS):
    qout += "\n" + qtempl.format(path=(str(f)+"_"+str(q)))
    outf.write(qout)

[B] template query stored in template.txt

declare
  %rest:path("test/{path}")
  %rest:GET
function _:f{path}() {{
   "{path}"
}};



Cheers,
Christian

[1] http://docs.basex.org/wiki/Options#PARSERESTXQ




On Thu, May 16, 2019 at 1:27 PM Marco Lettere  wrote:

Hello everyone,

we came accross an inconsistency which we are not able to find an
explanation for and giving up after having spent on it the whole morning...

  From a restxq we call an external REST service (InfluxDB) like shown in
[1]. This usually works as expected returning a response forged as in [2].

When the same RESTXQ is deployed in a much more complex application
(with several RESTXQ modules and REPO modules) the first call (which
takes a lot of time because it incurs the initial compiling delay)
returns [3] instead of [2]. From the second call onwards the result
stabilizes correctly to [2].

If I call any other RESTXQ function as a sort of warm-up (spending there
all the initial delay) [1] behaves correctly returning always [2].

Do you please have a clue on why this happens?

We noticed it for the first time because we started to use service jobs
which cause this access pattern ...

Thanks for any help.

[1]

module namespace test = 'test';

declare
%rest:path("test")
%rest:GET
function test:test() {
let $url :=
"http://localhost:8086/query?pretty=falsechunked=falsedb=nestwayq=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27;
let $results := http:send-request(, $url)
return $results
};

[2]

http://expath.org/ns/http-client;
status="200" message="OK">
  
  
  
  
  
  
  


  
  <_ type="object">
  
  <_ type="object">
  frontoffice
  
  <_>time
  <_>consumable
  <_>customer
  <_>guests
  <_>type
  <_>user
  
  
  <_ type="array">
<_>2019-05-10T21:55:00Z
  <_>access
<_>c9a83ce3-c938-4677-a0df-72bc96354509
  <_ type="number">0
  <_>starttimer
<_>d50de945-23ed-4b33-84e8-8ef533a96fb7
  
  
  
  
  
  



[3]

http://expath.org/ns/http-client;
status="200" message="OK">
  
  
  
  
  
  
  

map {
"results": [map {
  "series": [map {
"values": [["2019-05-10T21:55:00Z", "access",
"c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer",
"d50de945-23ed-4b33-84e8-8ef533a96fb7"]],
"name": "frontoffice",
"columns": ["time", "consumable", "customer", "guests", "type",
"user"]
  }]
}]
}





Re: [basex-talk] Optimizer and JSON/XML response

2019-05-16 Thread Marco Lettere

Just to add up ...

BaseX version is 9.0.2.

Moreover when running from a schedule service job the only way to avoid 
the issue described is to always first call another RESTXQ and then [1].
Which is somehow what I expect if services are run in their own 
process/jvm...
Despite the subject of the mail it looks like an issue in the RESTXQ 
parser/compiler rather than an optimizer issue...

M.


On 16/05/19 13:27, Marco Lettere wrote:

Hello everyone,

we came accross an inconsistency which we are not able to find an 
explanation for and giving up after having spent on it the whole 
morning...


From a restxq we call an external REST service (InfluxDB) like shown 
in [1]. This usually works as expected returning a response forged as 
in [2].


When the same RESTXQ is deployed in a much more complex application 
(with several RESTXQ modules and REPO modules) the first call (which 
takes a lot of time because it incurs the initial compiling delay) 
returns [3] instead of [2]. From the second call onwards the result 
stabilizes correctly to [2].


If I call any other RESTXQ function as a sort of warm-up (spending 
there all the initial delay) [1] behaves correctly returning always [2].


Do you please have a clue on why this happens?

We noticed it for the first time because we started to use service 
jobs which cause this access pattern ...


Thanks for any help.

[1]

module namespace test = 'test';

declare
  %rest:path("test")
  %rest:GET
function test:test() {
  let $url := 
"http://localhost:8086/query?pretty=falsechunked=falsedb=nestwayq=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27;

  let $results := http:send-request(, $url)
  return $results
};

[2]

http://expath.org/ns/http-client; 
status="200" message="OK">

    
    
    
    
    value="95b729c7-77ca-11e9-8065-"/>

    
    


    
    <_ type="object">
    
    <_ type="object">
    frontoffice
    
    <_>time
    <_>consumable
    <_>customer
    <_>guests
    <_>type
    <_>user
    
    
    <_ type="array">
<_>2019-05-10T21:55:00Z
    <_>access
<_>c9a83ce3-c938-4677-a0df-72bc96354509
    <_ type="number">0
    <_>starttimer
<_>d50de945-23ed-4b33-84e8-8ef533a96fb7
    
    
    
    
    
    



[3]

http://expath.org/ns/http-client; 
status="200" message="OK">

    
    
    
    
    value="7957030a-77cc-11e9-8066-"/>

    
    

map {
  "results": [map {
    "series": [map {
  "values": [["2019-05-10T21:55:00Z", "access", 
"c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer", 
"d50de945-23ed-4b33-84e8-8ef533a96fb7"]],

  "name": "frontoffice",
  "columns": ["time", "consumable", "customer", "guests", "type", 
"user"]

    }]
  }]
}





[basex-talk] Optimizer and JSON/XML response

2019-05-16 Thread Marco Lettere

Hello everyone,

we came accross an inconsistency which we are not able to find an 
explanation for and giving up after having spent on it the whole morning...


From a restxq we call an external REST service (InfluxDB) like shown in 
[1]. This usually works as expected returning a response forged as in [2].


When the same RESTXQ is deployed in a much more complex application 
(with several RESTXQ modules and REPO modules) the first call (which 
takes a lot of time because it incurs the initial compiling delay) 
returns [3] instead of [2]. From the second call onwards the result 
stabilizes correctly to [2].


If I call any other RESTXQ function as a sort of warm-up (spending there 
all the initial delay) [1] behaves correctly returning always [2].


Do you please have a clue on why this happens?

We noticed it for the first time because we started to use service jobs 
which cause this access pattern ...


Thanks for any help.

[1]

module namespace test = 'test';

declare
  %rest:path("test")
  %rest:GET
function test:test() {
  let $url := 
"http://localhost:8086/query?pretty=falsechunked=falsedb=nestwayq=SELECT+*+FROM+%22frontoffice%22+WHERE+time+%3E%3D+%272019-05-01T00%3A00%3A00Z%27+AND+time+%3C%3D+%272019-05-31T23%3A59%3A50Z%27;

  let $results := http:send-request(, $url)
  return $results
};

[2]

http://expath.org/ns/http-client; 
status="200" message="OK">

    
    
    
    
    value="95b729c7-77ca-11e9-8065-"/>

    
    


    
    <_ type="object">
    
    <_ type="object">
    frontoffice
    
    <_>time
    <_>consumable
    <_>customer
    <_>guests
    <_>type
    <_>user
    
    
    <_ type="array">
<_>2019-05-10T21:55:00Z
    <_>access
<_>c9a83ce3-c938-4677-a0df-72bc96354509
    <_ type="number">0
    <_>starttimer
<_>d50de945-23ed-4b33-84e8-8ef533a96fb7
    
    
    
    
    
    



[3]

http://expath.org/ns/http-client; 
status="200" message="OK">

    
    
    
    
    value="7957030a-77cc-11e9-8066-"/>

    
    

map {
  "results": [map {
    "series": [map {
  "values": [["2019-05-10T21:55:00Z", "access", 
"c9a83ce3-c938-4677-a0df-72bc96354509", 0.0e0, "starttimer", 
"d50de945-23ed-4b33-84e8-8ef533a96fb7"]],

  "name": "frontoffice",
  "columns": ["time", "consumable", "customer", "guests", "type", 
"user"]

    }]
  }]
}



Re: [basex-talk] BaseX 9.2: The Spring Edition

2019-04-17 Thread Marco Lettere

Yes, great question. We were asking this ourselves too.
M.

On 17/04/19 11:33, Andy Bunce wrote:


I am not clear what is meant by:
> - no need to import basex-api modules anymore!
Does this just mean all modules listed at 
http://docs.basex.org/wiki/Module_Library are statically bound?


/Andy





Re: [basex-talk] BaseX 9.2: The Spring Edition

2019-04-17 Thread Marco Lettere

Thanks! Great work as usual!
M.

On 16/04/19 16:10, Christian Grün wrote:

Dear all,

We are more than pleased to announce version 9.2 of BaseX, our XML
framework, database system and XQuery 3.1 processor:

   http://basex.org/

This time, we have put a special focus on performance improvements:

XQUERY PERFORMANCE
- comparisons will more often be rewritten to hash joins
- optimized access to subsequences and single sequence items
- consistent propagation of static types to evaluation results
- improved static typing of function items (maps, arrays)
- more aggressive pre-evaluation at compile time

STORAGE PERFORMANCE
- large databases will be opened faster
- regular storage offset will be computed at runtime

GENERAL XQUERY FEATURES
- no need to import basex-api modules anymore!
- Archive Module: archive:create-from, new options
- Database Module: directory-based access via db:dir
- Profiling Module: new functions (prof:gc, prof:runtime)
- Validation Module: assign XSD validation features
- WebSocket Module: query scheduling via ws:eval
- Utility Module: various new helper functions
- XSLT Module: support for XML Catalogs (thank you Liam)

NEW OPTIONS
- RESTXQERRORS: Suppress parsing errors at runtime
- FULLPLAN: comprehensive query plan information

FULLTEXT FEATURES
- stemmer for Arabic texts added

USER INTERFACES
- GUI: better support for latest JDK versions (incl. OpenJDK)
- DBA: revised search features in log panel

For a more comprehensive list of added and updated features. please
check out our documentation: http://docs.basex.org/.

We are looking forward to your feedback.

Have fun with the new release,
Your BaseX Team





Re: [basex-talk] concurrent performance issues?

2019-04-08 Thread Marco Lettere

Hi Michael,
there has been a lot of discussion related to this issue. The most 
significant result that has alway clearly emerged and that Christian is 
always evangelizing us about is that beeing DB -access strictly serial, 
querying in paralllel is causing only trouble unless the data is 
actually split onto different disks. accessing in parallel just causes 
the OS/HW TO jump around on the disk missing al the cache efficiency and 
optimization.
So, unless you have hardware that really supports parallel access, just 
prefer sequential disk access.


Hoffe dies hilft.

M.


On 08/04/19 16:05, Michael Perkonigg wrote:


Hallo,

ich versuche gerade, BaseX hier für eine zentrale XML-DB zu evaluieren.

Die DB ist read-only. Wir haben also keinerlei Updates.

Wir haben verschiedene XQuery-Queries(?), die abgesetzt werden, dabei 
ist aufgefallen, dass wenn ich mehrere Queries parallel starte die 
Antwortzeiten unverhätnismässig steigen.


Einzeln:

Query1: 500ms

Query2: 1200ms

Parallel brauchen beide zusammen dann zwischen 3 und 5 Sekunden.

Erstaunlicherweise sind die Antwortzeiten mit der Option PARALLEL=1 am 
geringsten, d.h. bei der Defaulteinstellung PARALLEL=8 (und auch schon 
bei PARALLEL=2) steigen die Antwortzeiten.


Ich frage mich nun, ob BaseX als Datenbank für eine verteilte 
Applikation mit zentraler DB überhaupt geeignet ist und ob ich 
vielleicht mit irgendwelchen Optionen BaseX für meine Zwecke 
optimieren kann.


Ganz theoretisch könnte die DB ja mit hundert parallelen Anfragen 
belastet werden.



--
mit freundlichen Grüßen





Re: [basex-talk] Hitting the Tail Recursion wall

2019-04-02 Thread Marco Lettere
If you need to join the resulting strings just use  well ... 
string-join ...


declare function local:topath($path){
  let $pathseg := tokenize($path, "/")
  let $pathsequence :=
    fold-left($pathseg, (), function($out, $segment){
  if($segment = "." or $segment = "") then $out
  else if($segment = "..") then $out[position() lt count($out)]
  else ($out, $segment)
    })
  return string-join($pathsequence, "/")
};

local:topath("/a/b/c/../../../g")

On 01/04/19 22:02, Andreas Mixich wrote:

Marco Lettere wrote onm 01.04.2019 at 18:01:


declare function local:topath($path){  let $pathseg := tokenize($path,
"/")  return    fold-left($pathseg, (), function($out, $segment){
if($segment = "." or $segment = "") then $out  else if($segment =
"..") then $out[position() lt count($out)]  else ($out,
$segment)    })}; local:topath("/a/b/c/../../../g")

Beautiful and very close, except for a minor caveat: the "/" are needed,
to reconstruct the absolute path-part from the relative.
I played around and tried to place some "/" to your function, but all
variants placed some "/" wrong or twice.

Last but not least: You asked for the use case, which is described on
https://tools.ietf.org/html/rfc3986#section-5.2.4 (note also the
sequences shown after the description of the steps)


 5.2.4 <https://tools.ietf.org/html/rfc3986#section-5.2.4>.
 Remove Dot Segments



The pseudocode also refers to a "remove_dot_segments" routine for
interpreting and removing the special "." and ".." complete path
segments from a referenced path.  This is done after the path is
extracted from a reference, whether or not the path was relative, in
order to remove any invalid or extraneous dot-segments prior to
forming the target URI.  Although there are many ways to accomplish
this removal process, we describe a simple method using two string
buffers.

1.  The input buffer is initialized with the now-appended path
components and the output buffer is initialized to the empty
string.

2.  While the input buffer is not empty, loop as follows:

A.  If the input buffer begins with a prefix of "../" or "./",
then remove that prefix from the input buffer; otherwise,

B.  if the input buffer begins with a prefix of "/./" or "/.",
where "." is a complete path segment, then replace that
prefix with "/" in the input buffer; otherwise,

C.  if the input buffer begins with a prefix of "/../" or "/..",
where ".." is a complete path segment, then replace that
prefix with "/" in the input buffer and remove the last
segment and its preceding "/" (if any) from the output
buffer; otherwise,

D.  if the input buffer consists only of "." or "..", then remove
that from the input buffer; otherwise,

E.  move the first path segment in the input buffer to the end of
the output buffer, including the initial "/" character (if
any) and any subsequent characters up to, but not including,
the next "/" character or the end of the input buffer.

3.  Finally, the output buffer is returned as the result of
remove_dot_segments.

Note that dot-segments are intended for use in URI references to
express an identifier relative to the hierarchy of names in the base
URI.  The remove_dot_segments algorithm respects that hierarchy by
removing extra dot-segments rather than treat them as an error or
leaving them to be misinterpreted by dereference implementations.

The following illustrates how the above steps are applied for two
examples of merged paths, showing the state of the two buffers after
each step.

   STEP   OUTPUT BUFFER INPUT BUFFER

1 : /a/b/c/./../../g
2E:   /a/b/c/./../../g
2E:   /a/b  /c/./../../g
2E:   /a/b/c/./../../g
2B:   /a/b/c/../../g
2C:   /a/b  /../g
2C:   /a/g
2E:   /a/g

   STEP   OUTPUT BUFFER INPUT BUFFER

1 <https://tools.ietf.org/html/rfc3986#section-1> : 
mid/content=5/../6
2E:   mid   /content=5/../6
2E:   mid/content=5 /../6
2C:   mid   /6
2E:   mid/6









Re: [basex-talk] Hitting the Tail Recursion wall

2019-04-01 Thread Marco Lettere

Hi Andreas,
I don't know whether I correctly understood you use-case but what about 
going with hof functions [1]?

Maybe your code could turn to something as simple as

declare function local:topath($path){
  let $pathseg := tokenize($path, "/")
  return
    fold-left($pathseg, (), function($out, $segment){
  if($segment = "." or $segment = "") then $out
  else if($segment = "..") then $out[position() lt count($out)]
  else ($out, $segment)
    })
};
local:topath("/a/b/c/../../../g")

Regards,
Marco.

[1] http://docs.basex.org/wiki/Higher-Order_Functions

On 01/04/19 15:49, Andreas Mixich wrote:

I think, I am finding it...

I replaced the first expression with

`prof:dump(concat("in: ", $path, " out: ",
string-join(array:flatten($out,`

and this gives me more info. So I may be able to solve this alone.





Re: [basex-talk] Quick Windows issue: Invalid reply on port 1984, works fine on port 8984

2019-03-29 Thread Marco Lettere

Hi Jim,

I think port 1984 is not meant to be connected to over HTTP. It's a pure 
socket connection that ships BaseX internal client-server protocol.
When you connect to that port you probably have to use the Python client 
API in order to get your BaseX COMMANDS (not HTTP requests) serialized 
over the wire.


Hope this helps [cit.].

Regards,
M.

On 28/03/19 23:28, Jim Salmons wrote:

Hello BaseX folk,

I have just installed version 9.1.2 and am starting to learn BaseX in
support of my post-cancer #PayItForward Bonus Round #CitizenScientist
research in #DigitalHumanities and #MachineLearning. My goal is to "cut out
the middleman" of writing intermediate JSON and XML fragment files in order
to update the #MAGAZINEgts ground-truth storage file that I am developing
for a digital collection of Softalk Apple magazine at the Internet Archive.
With BaseX and its Python interface, I want to write new and updated
document structure and content depiction metadata directly to a local copy
of this ground-truth master file which will then be easily popped out of the
database as an XML file for uploading to the Softalk Apple collection at the
Internet Archive.

I hit a bump, reported to the Github Issue queue, where my Windows 10
installation can not open the BaseX GUI exe launcher. The work-around, as
noted in my issue report, is to use the Getting Started advice to execute
the jar directly. By creating a desktop icon that launches the execuable jar
in the installation folder, the GUI launches as expected.

With that minor issue behind me, I am wondering why my attempts to connect
to the server via localhost:1984 port does not work while everything is fine
and dandy when hitting port 8984. Port 1984 returns an invalid HTTP
response. (Note: Other ports that are not assigned to anything else return a
connection refused, so this invalid response is not the same as the server
not being there.)

If I keep a Powershell window open with the server running, every once in a
while (after several minutes or more) the server throws this error:

java.lang.NullPointerException
 at org.basex.core.Context.perm(Context.java:252)
 at org.basex.core.Command.run(Command.java:250)
 at org.basex.core.Command.run(Command.java:143)
 at org.basex.core.cmd.Close.close(Close.java:54)
 at org.basex.server.ClientListener.close(ClientListener.java:232)
 at org.basex.BaseXServer$1.run(BaseXServer.java:157)
 at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
 at java.base/java.util.TimerThread.run(Timer.java:506)

which may be related to port 1984 not responding, a firewall issue perhaps?
Honestly, I don't see anything that looks like a BaseX process or service
running in Task Manager (other than a generic 'Java(TM) Platform SE binary'
process, and there is nothing that looks related in the Windows Firewall
entries. I'm still investigating this here, but if anyone knows what is
akimbo, I would greatly appreciate your advice.

Thank you... and looking forward to being a new BaseX community member. :-)

Happy-Healthy Vibes,
-: Jim Salmons :-

 Jim Salmons
     Twitter: @Jim_Salmons

     https://www.researchgate.net/profile/Jim_Salmons (my #CitizenScientist
profile)
 www.medium.com/@Jim_Salmons/ (my mostly #DigitalHumanities writing)






Re: [basex-talk] Serialization issue with HTTP response

2019-03-08 Thread Marco Lettere

Great! Thanks!

On 08/03/19 13:16, Christian Grün wrote:

Hi Marco,

Sorry for letting you wait. Specializations of the json mimetype will
now be detected as well [1,2].

Cheers,
Christian

[1] 
https://github.com/BaseXdb/basex/commit/a18a092b11a5240075b081ffd301014f0dc00edf
[2] http://files.basex.org/releases/latest/


On Tue, Jan 15, 2019 at 11:35 AM Marco Lettere  wrote:

Hi all,
we have to deal with a third-party REST service which in case of error 
conditions returns this mime type: application/problem+json; charset=utf-8.

I wrote this RestXQ [1] to mock it. Just copy it into restxq.xqm...

When I call it like [2] I get [XPTY0004] Cannot convert xs:base64Binary to xs:string: 
"IntcImFcIjpcImJcIn0i". This is because the response is seen as a binary and 
this can be double checked calling with [3].
I felt (maybe wrongly) that specializations of the mimetype had been faced in 
BaseX implementation of http:request somewhere back in time but I couldn't find 
anything in the mail archives.

The strange thing however is that I get the same error when calling like [4] 
with override-mime-type set to text/json or application/json which instead I 
was expecting to work.

Is the behaviour in this scenario expected or does this hide some inconsistency?

Thanks, as usual, for the support.

Regards,

Marco.

[1]
declare%rest:path("mc/dummy")
%rest:GET
%output:method("json")
function page:dummy() {
   
 
   
 
   ,
   '{"a":"b"}'
};

[2]
json:parse(
   http:send-request(,"http://localhost:8984/mc/dummy;)[2]
)

[3]
json:parse(
   convert:binary-to-string(
 http:send-request(,"http://localhost:8984/mc/dummy;)[2]
   )
)

[4]
json:parse(
   http:send-request(,"http://localhost:8984/mc/dummy;)[2]
)





Re: [basex-talk] Serialization issue with HTTP response

2019-03-06 Thread Marco Lettere

Hi Christian,
I'd like to kindly ask whether there has been some follow-up on this 
matter...

Thanks,
M.

On 16/01/19 16:27, Christian Grün wrote:

I hope I’ll be able to test this tomorrow. And your mail is yet
another motivation to get the upcoming version 2 of the HTTP Client
finished (http://expath.github.io/expath-cg/specs/http-client-2).



On Tue, Jan 15, 2019 at 11:35 AM Marco Lettere  wrote:

Hi all,
we have to deal with a third-party REST service which in case of error 
conditions returns this mime type: application/problem+json; charset=utf-8.

I wrote this RestXQ [1] to mock it. Just copy it into restxq.xqm...

When I call it like [2] I get [XPTY0004] Cannot convert xs:base64Binary to xs:string: 
"IntcImFcIjpcImJcIn0i". This is because the response is seen as a binary and 
this can be double checked calling with [3].
I felt (maybe wrongly) that specializations of the mimetype had been faced in 
BaseX implementation of http:request somewhere back in time but I couldn't find 
anything in the mail archives.

The strange thing however is that I get the same error when calling like [4] 
with override-mime-type set to text/json or application/json which instead I 
was expecting to work.

Is the behaviour in this scenario expected or does this hide some inconsistency?

Thanks, as usual, for the support.

Regards,

Marco.

[1]
declare%rest:path("mc/dummy")
%rest:GET
%output:method("json")
function page:dummy() {
   
 
   
 
   ,
   '{"a":"b"}'
};

[2]
json:parse(
   http:send-request(,"http://localhost:8984/mc/dummy;)[2]
)

[3]
json:parse(
   convert:binary-to-string(
 http:send-request(,"http://localhost:8984/mc/dummy;)[2]
   )
)

[4]
json:parse(
   http:send-request(,"http://localhost:8984/mc/dummy;)[2]
)





Re: [basex-talk] RESTXQ and regexp

2019-02-25 Thread Marco Lettere

Thanks for your suggestion Liam!
We've tried this out on the current deployment which involves version 
8.6.3 and it didn't fix our specific use case.
My idea here (I'm not sure to make the correct assumption though) is 
that the proposition "if multiple candidate functions ar eleft over" 
stated in the docs is not holding here.
It rather looks as if BaseX is considering only the more generic one to 
match.

Regards,
Marco.

On 23/02/19 23:33, Liam R. E. Quin wrote:

On Fri, 2019-02-22 at 12:05 +0100, Marco Lettere wrote:


(: Matches anything followed by /input :)
declare %rest:path("app/{$path=.+}/input")
function page:inputs($path) { ... };

(: Matches all other all paths starting with "app/" :)
declare %rest:path("app/{$path=.+}")
function page:others($path) { ... };

Can you use content negotiation/quality to say the /input one is
preferred when both match? E.g. %rest:produces("*/*;qs=0.8") on the
page:others function and */*;qs=1.0 on the input ont?

Liam






Re: [basex-talk] BaseX-Talk Digest, Vol 110, Issue 29

2019-02-25 Thread Marco Lettere

Hi Shaun,
I agree on your suggestions related to the regexp.
The thing is we _have_ tried to achieve a more fine grained filter by 
matching in the $path related segment app/"anything but the word 
input"/input.
Unfortunately according to my knowledge this requires fiddling with 
positive lookahead expressions. We haven't been able to make them work 
in RESTXQ.
We hoped that being /app/"anything"/input/ more specific than 
/app/"anything"/, the former would ha been preferred. But it doesn't 
seem to work like that.

Regards,
Marco.


On 23/02/19 22:11, Shaun Flynn wrote:

Hello there Marco,

This might be down to the choice of regex.

Assume $path = "path"

app/{$path=.+}

Will match

"app/path=1234"

And

"app/path=1234/input"

Which I think would be matched by ".+" part of the expression, as 
"/input" would match ".+".


Kind regards

On Sat, 23 Feb 2019, 11:00 , 
<mailto:basex-talk-requ...@mailman.uni-konstanz.de>> wrote:


Send BaseX-Talk mailing list submissions to
basex-talk@mailman.uni-konstanz.de
<mailto:basex-talk@mailman.uni-konstanz.de>

To subscribe or unsubscribe via the World Wide Web, visit
https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
or, via email, send a message with subject or body 'help' to
basex-talk-requ...@mailman.uni-konstanz.de
<mailto:basex-talk-requ...@mailman.uni-konstanz.de>

You can reach the person managing the list at
basex-talk-ow...@mailman.uni-konstanz.de
<mailto:basex-talk-ow...@mailman.uni-konstanz.de>

When replying, please edit your Subject line so it is more specific
than "Re: Contents of BaseX-Talk digest..."


Today's Topics:

   1. RESTXQ and regexp (Marco Lettere)


------

Message: 1
Date: Fri, 22 Feb 2019 12:05:29 +0100
From: Marco Lettere mailto:m.lett...@gmail.com>>
To: BaseX mailto:basex-talk@mailman.uni-konstanz.de>>
Subject: [basex-talk] RESTXQ and regexp
Message-ID: <3f71e0bc-e16d-6c4d-6ee8-246a7cc75...@gmail.com
<mailto:3f71e0bc-e16d-6c4d-6ee8-246a7cc75...@gmail.com>>
Content-Type: text/plain; charset=utf-8; format=flowed

Hello all,

according to the docs this should be possible:

(: Matches all paths with "app" as first, a number as second, and
"order" as third segment :)
declare %rest:path("app/{$code=[0-9]+}/order")
 ? function page:order($full-path) { ... };

(: Matches all other all paths starting with "app/" :)
declare %rest:path("app/{$path=.+}")
 ? function page:others($path) { ... };

And it actually is... We are currently in the very unfortunate
situation
where we need the following:

(: Matches anything followed by /input :)
declare %rest:path("app/{$path=.+}/input")
 ? function page:inputs($path) { ... };

(: Matches all other all paths starting with "app/" :)
declare %rest:path("app/{$path=.+}")
 ? function page:others($path) { ... };

Unfortunately this makes all the calls to be redirected to the
page:others function ... is there a motivation behind it or could
it be
possible to work around that?

Thanks,

Marco.



End of BaseX-Talk Digest, Vol 110, Issue 29
***





[basex-talk] RESTXQ and regexp

2019-02-22 Thread Marco Lettere

Hello all,

according to the docs this should be possible:

(: Matches all paths with "app" as first, a number as second, and 
"order" as third segment :)

declare %rest:path("app/{$code=[0-9]+}/order")
  function page:order($full-path) { ... };

(: Matches all other all paths starting with "app/" :)
declare %rest:path("app/{$path=.+}")
  function page:others($path) { ... };

And it actually is... We are currently in the very unfortunate situation 
where we need the following:


(: Matches anything followed by /input :)
declare %rest:path("app/{$path=.+}/input")
  function page:inputs($path) { ... };

(: Matches all other all paths starting with "app/" :)
declare %rest:path("app/{$path=.+}")
  function page:others($path) { ... };

Unfortunately this makes all the calls to be redirected to the 
page:others function ... is there a motivation behind it or could it be 
possible to work around that?


Thanks,

Marco.



Re: [basex-talk] Global lock = false and parallel update processes to different DBs

2019-02-06 Thread Marco Lettere
Hi France,
I recall once I've been successful in generating xquery strings by patching
the database name into it and then processing it with xquery:eval.
Might this be anche approach for you?
M.

Il giorno mer 6 feb 2019, 09:12 France Baril 
ha scritto:

> Irsh, we have 52 languages and all our system is based on being able to
> work with any language and let clients add/remove languages without having
> to call developers. I can't imagine the domino effect of having to build a
> shell function per language per process that access the DB.
>
> Plus as we are running batch processes, I think we'll just run out of
> memory.
>
> I'm thinking one function like this per language is what you propose :
>
> rest-path /base/filter-es-us()
> function filter-es-us {
> let $src-db = db:open(es-us)
> let $results := apply-non-updating-processes($src-db)...  where result is
> a map of (filename, xml)
> return
>   for $result in $results
>   return  db:replace('staging-es-us', $results)
> };
>
> apply-non-updating-processes($src-db){
> map:merge(
>  for $file in $src-db/*
>  res= do x
>  return map:entry ($file/base-uri, res)
> };
>
>
> Since we run batch processes I'm also thinking we'll run out of memory
> with processes like that... or maybe we need to split also small functions
> so each tiny update is in its own function... then maintaiing functions for
> 52 languages becomes even harder... or I add an extra layer of abstraction
> and build the .xqm functions dynamically based on a central code base and
> the dynamic language names... hmmm
>
> I'm thinking out loud here trying to find my way outside of dynamic
> names... but static naming of databases doesn't sound like a good idea in
> our case. Dynamic naming is at the core of our approach... or maybe I'm so
> laced in it that I can't see the easy way in?
>
>
>
>
>
> On Mon, Feb 4, 2019 at 11:46 AM Christian Grün 
> wrote:
>
>> Hi France,
>>
>> > I noticed that the latest version of BaseX lost this feature and
>> nothing seems to replace it. I'm trying to improve performance of batch
>> processes and I was counting on that feature a lot. Any change it will come
>> back or that something equivalent will come?
>>
>> With BaseX 9, we removed the classical GLOBALLOCK option (i.e.,
>> GLOBALLOCK = false is standard now).
>>
>> > get db:open($lang)/*
>> > process
>> > save to db:open('staging-' || $lang)
>>
>> The name of your database may be specified as static string in your
>> query (no matter if you use BaseX 8 or 9):
>>
>>   get db:open('de')/*
>>   process
>>   save to db:open('staging-de')
>>
>> Did you try this already?
>> Christian
>>
>
>
> --
> France Baril
> Architecte documentaire / Documentation architect
> france.ba...@architextus.com
>


Re: [basex-talk] BaseX/GUI v9.1.2 memory use

2019-01-23 Thread Marco Lettere
Same for us over here. The ability to follow symlinks is a very powerful 
feature that we use to externalize folders (data, restxq for instance). 
So please don't remove it altogether!

M.

On 22/01/19 23:50, Graydon Saunders wrote:
I've been handling updates by making data/ a symbolic link to a data 
directory that's a sibling of the basex directory.  (Move the old, 
unpack the new, go into new and replace data/ with a symbolic link up 
and over.)


Would hate to see that stop working.

On Tue, Jan 22, 2019, 17:36 Christian Grün > wrote:


Good to hear that! I can’t recollect that something particular has
changed in version 9.1.2, regarding the scanning of project files,
but I’ll have some thoughts how we can trace and interrupt such
loops (or ignore symbolic links instead).



Am Di., 22. Jan. 2019, 23:22 hat Bridger Dyson-Smith
mailto:bdysonsm...@gmail.com>> geschrieben:

Glad that helped :)

I see this when I start from a fresh install vs expanding the
ZIP into the same directory.

On Tue, Jan 22, 2019, 5:17 PM Rick Graham
mailto:rickhg1...@gmail.com> wrote:

Thanks Bridger!

Indeed, I quit basexgui and manually edited .basexgui to
set the project directory to a newly created empty
directory.  basexgui seems normal/stable after that.

I rarely, as in almost never, use wine but I didn't have
this issue with previous versions of BaseX.  Something
seems unexpected here.


On Tue, Jan 22, 2019 at 11:04 PM Bridger Dyson-Smith
mailto:bdysonsm...@gmail.com>> wrote:

Hi Rick, et al,
I think (but am not 100% sure) that the GUI defaults
to looking through your home directory on startup. So,
somewhere in `~/rick/.wine/dosdevices/...` you have
symbolic links that are looped.

I think you might be able to circumvent this problem
by finding `.basexgui` - it would probably be close to
wherever you started the GUI from on your filesystem.
I think you can edit some of the PATHS there and that
may help?

Again, I'm not sure. HTH!
Best,
Bridger

On Tue, Jan 22, 2019 at 4:56 PM Rick Graham
mailto:rickhg1...@gmail.com>>
wrote:

The command-line seemed to be operating normally.

What exactly is/are my project directories?

I attached to the running GUI instance `strace -f
-e trace=stat -p 13368` and it has infinite
repetitions of:

[pid 13436]

stat("/home/rick/.wine/dosdevices/z:/sys/class/thermal/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone2/subsystem/thermal_zone0/device/subsystem/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:75/subsystem/devices/PNP0C0A:02",
0x7f7beb2796e0) = -1 ELOOP (Too many levels of
symbolic links)


What's going on here?

On Tue, Jan 22, 2019 at 10:21 PM Christian Grün
mailto:christian.gr...@gmail.com>> wrote:

at

org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:167)
at

org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)
at

org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)
at

org.basex.gui.view.project.ProjectFiles.add(ProjectFiles.java:173)


Looks like a endless loop that is caused by
parsing the files in your project directory.
Do you possibly have any symbolic links?

Can you reproduce the problem with a
completely fresh BaseX zip archive?






Re: [basex-talk] Documentation suggestion

2019-01-17 Thread Marco Lettere

One more ... the signature of hmac function at [1] looks messed up...

[1] http://docs.basex.org/wiki/Cryptographic_Module#crypto:hmac

On 16/01/19 16:42, Christian Grün wrote:

I had one but lost it like 99% of my account credentials ... :-(

Sorry to hear that. – I’ll be happy to enable another one for you.



M.

On 16/01/19 16:25, Christian Grün wrote:

Hi Marco,


I suggest to add an explicit statement of which is the default value of
the combine option of map:merge.

Done. If you are interested, feel free to get a little account for our Wiki!

Christian






Re: [basex-talk] Documentation suggestion

2019-01-16 Thread Marco Lettere

Thanks.
I had one but lost it like 99% of my account credentials ... :-(
M.

On 16/01/19 16:25, Christian Grün wrote:

Hi Marco,


I suggest to add an explicit statement of which is the default value of
the combine option of map:merge.

Done. If you are interested, feel free to get a little account for our Wiki!

Christian





Re: [basex-talk] Serialization issue with HTTP response

2019-01-16 Thread Marco Lettere



Just WoW! Looks great!
M.

On 16/01/19 16:27, Christian Grün wrote:

I hope I’ll be able to test this tomorrow. And your mail is yet
another motivation to get the upcoming version 2 of the HTTP Client
finished (http://expath.github.io/expath-cg/specs/http-client-2).



On Tue, Jan 15, 2019 at 11:35 AM Marco Lettere  wrote:

Hi all,
we have to deal with a third-party REST service which in case of error 
conditions returns this mime type: application/problem+json; charset=utf-8.

I wrote this RestXQ [1] to mock it. Just copy it into restxq.xqm...

When I call it like [2] I get [XPTY0004] Cannot convert xs:base64Binary to xs:string: 
"IntcImFcIjpcImJcIn0i". This is because the response is seen as a binary and 
this can be double checked calling with [3].
I felt (maybe wrongly) that specializations of the mimetype had been faced in 
BaseX implementation of http:request somewhere back in time but I couldn't find 
anything in the mail archives.

The strange thing however is that I get the same error when calling like [4] 
with override-mime-type set to text/json or application/json which instead I 
was expecting to work.

Is the behaviour in this scenario expected or does this hide some inconsistency?

Thanks, as usual, for the support.

Regards,

Marco.

[1]
declare%rest:path("mc/dummy")
%rest:GET
%output:method("json")
function page:dummy() {
   
 
   
 
   ,
   '{"a":"b"}'
};

[2]
json:parse(
   http:send-request(,"http://localhost:8984/mc/dummy;)[2]
)

[3]
json:parse(
   convert:binary-to-string(
 http:send-request(,"http://localhost:8984/mc/dummy;)[2]
   )
)

[4]
json:parse(
   http:send-request(,"http://localhost:8984/mc/dummy;)[2]
)





[basex-talk] Documentation suggestion

2019-01-15 Thread Marco Lettere

Hi Christian,

I suggest to add an explicit statement of which is the default value of 
the combine option of map:merge.


Currently the only reference is in the Change notes at the bottom of the 
page.


Thanks!

Marco.



[basex-talk] Serialization issue with HTTP response

2019-01-15 Thread Marco Lettere

Hi all,
we have to deal with a third-party REST service which in case of error 
conditions returns this mime type: /*application/problem+json; 
charset=utf-8*/.


I wrote this RestXQ [1] to mock it. Just copy it into restxq.xqm...

When I call it like [2] I get /*[XPTY0004] Cannot convert 
xs:base64Binary to xs:string: "IntcImFcIjpcImJcIn0i"*/. This is because 
the response is seen as a binary and this can be double checked calling 
with [3].
I felt (maybe wrongly) that specializations of the mimetype had been 
faced in BaseX implementation of http:request somewhere back in time but 
I couldn't find anything in the mail archives.


The strange thing however is that I get the same error when calling like 
[4] with override-mime-type set to text/json or application/json which 
instead I was expecting to work.


Is the behaviour in this scenario expected or does this hide some 
inconsistency?


Thanks, as usual, for the support.

Regards,

Marco.

[1]
declare%rest:path("mc/dummy")
%rest:GET
%output:method("json")
function page:dummy() {
  
    
  

    
  ,
  '{"a":"b"}'
};

[2]
json:parse(
  http:send-request(method="get"/>,"http://localhost:8984/mc/dummy;)[2]

)

[3]
json:parse(
  convert:binary-to-string(
    http:send-request(method="get"/>,"http://localhost:8984/mc/dummy;)[2]

  )
)

[4]
json:parse(
  http:send-request(override-media-type="application/json"/>,"http://localhost:8984/mc/dummy;)[2]

)


Re: [basex-talk] BaseX 9.1.1

2018-12-14 Thread Marco Lettere

Yeah! A Christmas edition!
Thank you all for this!
M.

On 14/12/18 11:31, Christian Grün wrote:

Hi all,

we are glad to provide you with version 9.1.1 of BaseX! It’s actually
more than just a maintenance release:

  XQuery:
  - Comprehensive rewritings of positional predicates and functions
  - Higher-order functions: Improved type inference
  - Improved rewriting of context-based and/or nested predicates

  Java Bindings:
  - Faster access to and evaluation of Java functions and variables
  - Improved pre-selection of function candidates at compile time
  - Better error messages (incl. function arity and similar names)

  DBA:
  - Settings: user-defined pattern for ignoring log entries
  - Login: pass on URL query strings

  Minor improvements:
  - Import: detect epub files as ZIP archives
  - Digest Authentication: No delay after first request
  - GUI, Preferences: user-defined choice of XML suffixes

The new version is online: http://basex.org/

Looking forward to your feedback as usual.
Have a nice time,
Christian





Re: [basex-talk] restxq

2018-11-09 Thread Marco Lettere
It looks like your basexhttp server is pointing to a different directory 
than the one you expect.
This might happen from time to time depending on type of installation 
env variables and things like that.

M.

On 09/11/18 13:30, Christian Grün wrote:

…difficult to tell. Could you please provide us with a minimized
version and tell us the exact steps how to proceed (1. download
basex91.zip, 2. unzip, etc.)?
On Fri, Nov 9, 2018 at 1:03 PM Giuseppe G. A. Celano
 wrote:

I am trying to make a RESTXQ webservice I created with BaseX 8.3 available in Basex 9.2. I simply 
copied my "file.xqm" into the webapp folder but, when I type the path of a function 
contained in it, it does not work ("No function found that matches the request."). 
Strangely enough, when I minimally modify restxq.xqm, the modifications do not apply either. Any 
idea about what the problem might be? Thanks.





Re: [basex-talk] BaseX 9.1: The Autumn Edition

2018-10-31 Thread Marco Lettere

CHEERS!

M.

On 31/10/18 17:34, Christian Grün wrote:

Dear XML and XQuery aficionados,

It’s been exactly 5 months ago when BaseX 9 was released, and we are
happy to announce version 9.1 of our XML framework, database system
and XQuery 3.1 processor! The latest release is online:

   http://basex.org

The most exciting addition is support for WebSockets, which enable you
to do bidirectional (full-duplex) client/server communication with
XQuery web applications:

   http://docs.basex.org/wiki/WebSockets

Moreover, we have added convenient syntax extensions (ternary if,
Elvis operator, if without else) to XQuery. Some of them may be made
available in other implementations of XQuery as well (we’ll keep you
updated):

   http://docs.basex.org/wiki/XQuery_Extensions#Expressions

Other new features are as follows:

XQuery:
- set local locks via pragmas and function annotations
- Database Module: faster processing of value index functions
- Jobs Module: record and return registration times
- ENFORCEINDEX option: support for predicates with dynamic values
- Update Module, update:output: support for caching maps and arrays

GUI:
- Mac, Windows: Improved rendering support for latest Java versions
- XQuery editor: choose and display current query context

Visit http://docs.basex.org to get more information on the added features.

Your feedback is welcome! Have fun,

Christian
BaseX Team





Re: [basex-talk] Sir, when basex 9.1, please ;-)

2018-10-23 Thread Marco Lettere

Yes Christian,
it works with ws:emit() and I also understand the point now.
This embedded support for websockets allows us to reduce our code base 
in some use cases by several hundreds lines of code.
Not speaking about dependency management and configuration tricks to be 
documented.

Great job!
M.

On 22/10/18 13:36, Christian Grün wrote:

Hi Marco,

I’m glad to you are gathering some experience with the WebSocket facility.


I get the following error [2] as return to my HTTP POST and, ca va sans
dire, nothing on the websocket.

The function bound to the "/dataprovider" is a simple RESTXQ function.
As such, it isn’t attached to a WebSocket id. One of the reasons is
that a client who’s using RESTXQ may not necessarily have a WebSocket
connection, or there can also be multiple WebSockets per client.

If you want to send your result to all WebSockets – including the
client that called the dataprovider – you can simply use ws:emit(). If
your use case is complex enough to have a WebSocket connection and
simultaneous RESTXQ requests in a single browser tab, you could store
the WebSocket id(s) of your client as HTTP Session attribute, and
access these ids from the RESTXQ code.


BTW, even if not stated in the path annotation, an extra /ws needs to be
prefixed to the url used for JS' WebSocket constructor. Personally I'd
prefer to keep things explicit and put it in the annotation too.

I have just revised our documentation, and I hope it’s fairly complete
now. In the Annotations Section, you will find a hint to the "ws/"
path.

The reason why the path is omitted in XQuery is that the web server
takes care of the path resolution. If the default path is changed in
the web.xml file, it would need to be changed in all XQuery
applications as well. The same applies to RESTXQ: If a prefix is used
in the configuration, there won’t be a need to change your path
annotations.

Hope this helps,
Christian

[1] http://docs.basex.org/wiki/WebSockets



Thanks for your support and thanks Maximilian for the lift.

[1]
module namespace dp = 'urn:nubisware:datarouter';

import module namespace ws = 'http://basex.org/modules/ws';

declare
%rest:path('/dataprovider')
%rest:POST("{$data}")
%output:method('json')
function dp:route($data as node()) {
ws:send(json:serialize($data), ws:ids()[. != ws:id()])
};

declare
%ws:connect('/dataprovider')
function dp:connect() as empty-sequence() {
()
};

declare
%ws:close('/dataprovider')
function dp:close() as empty-sequence() {
()
};

[2]
Stopped at /home/lettere/tmp/basex/webapp/dataprovider/dataprovider.xqm,
11/53:
[basex:ws] WebSocket connection required.

On 18/10/18 18:33, Christian Grün wrote:

Sir, doing our best ;)

We believe that BaseX 9.1 pretty soon (until end of October).

For everyone who is interested in giving us some feedback on the new
WebSocket feature… Thank you in advance! 90% of the documentation is
finalized:

http://docs.basex.org/wiki/WebSockets

Best,
Christian






Re: [basex-talk] Sir, when basex 9.1, please ;-)

2018-10-22 Thread Marco Lettere

Hi,
I'm just hijacking Maximilian's email here to post the following test I 
wanted to do for experimenting with BaseX and Websockets.
I wrote the code at [1] with the intent to open a RESTXQ entrypoint that 
receives a JSON via POST and the broadcastst it to all connecte WS clients.
I might be using ws:broadcast or the naive version as shown but still I 
get the following error [2] as return to my HTTP POST and, ca va sans 
dire, nothing on the websocket.

What am I doing wrong?

BTW, even if not stated in the path annotation, an extra /ws needs to be 
prefixed to the url used for JS' WebSocket constructor. Personally I'd 
prefer to keep things explicit and put it in the annotation too.


Thanks for your support and thanks Maximilian for the lift.

[1]
module namespace dp = 'urn:nubisware:datarouter';

import module namespace ws = 'http://basex.org/modules/ws';

declare
  %rest:path('/dataprovider')
  %rest:POST("{$data}")
  %output:method('json')
function dp:route($data as node()) {
  ws:send(json:serialize($data), ws:ids()[. != ws:id()])
};

declare
  %ws:connect('/dataprovider')
function dp:connect() as empty-sequence() {
  ()
};

declare
  %ws:close('/dataprovider')
function dp:close() as empty-sequence() {
  ()
};

[2]
Stopped at /home/lettere/tmp/basex/webapp/dataprovider/dataprovider.xqm, 
11/53:

[basex:ws] WebSocket connection required.

On 18/10/18 18:33, Christian Grün wrote:

Sir, doing our best ;)

We believe that BaseX 9.1 pretty soon (until end of October).

For everyone who is interested in giving us some feedback on the new
WebSocket feature… Thank you in advance! 90% of the documentation is
finalized:

   http://docs.basex.org/wiki/WebSockets

Best,
Christian





Re: [basex-talk] Tomcat & services

2018-10-15 Thread Marco Lettere

On 15/10/2018 17:56, Christian Grün wrote:

Thanks for your summary on the behavior of Tomcat.

> In general we think that when working with tomcat one usually has to 
link an external dbpath in order to avoid getting his data removed 
when redeploying the webapp. The requirement to put jobs.xml inside 
the datapath somehow breaks the atomicity of a deployable webapp.
> A workaround could be to have the jobs.xml somehow redefined as a 
web.xml variable. Might this be possible?


To understand your use case better:
1. Do you currently define an external database path when working with 
Tomcat?
Yes. It's necessary. Otherwise, for each redeploy of the webapp, the 
data is lost.
2. Would you prefer to have a servlet-specific jobs.xml file and an 
external database directory, or should both of them be placed 
internally or externally?
I think there is nothing against how dbpath is handled now. One has just 
to uncomment the dbpath variable inside the web.xml of the webapps that 
require persistent data and change it. That's good enough for us.
At the same time the unfortunate consequence is that jobs.xml, which to 
our feeling is something that lives with code and not with data, has to 
be copied manually to the external dbpath in scenarios where you 
redeploy a webapp after some fixes or tuning. So our suggestion is to 
have something like a "jobs" variable in web.xml that points to an 
servlet internal jobs.xml (maybe in the WEB-INF folder?) to be used 
instead of the default one.


In any case I don't know if this solves the current misbehaviour we have 
with tomcat which simply and rudely ignores the content of jobs.xml 
altogether.


I hope things are slightly clearer now.
M.





Just a guess, but maybe it would generally be a good idea to be able 
to define an external database directory for Tomcat? Or would you 
indeed like to have servlet-specific databases removod if



On Mon, Oct 15, 2018 at 5:47 PM Marco Lettere <mailto:m.lett...@gmail.com>> wrote:


Ok, actually on my personal PC jobs are correctly listed also with
basexhttp (need to investigate why on my colleagues PC this
doesn't work).

Back to the more important issue...
I put the following jobs.xml[1] file in the data folder and get
[2] as first lines of log in dba after reboot.
I put the same file into the "dbpath" folder of my app running as
tomcat webapp but the log looks like [3].

The feeling is that services are scheduled only in accordance to
the startup of the http server which doesn't occur when running
inside Tomcat.

In general we think that when working with tomcat one usually has
to link an external dbpath in order to avoid getting his data
removed when redeploying the webapp. The requirement to put
jobs.xml inside the datapath somehow breaks the atomicity of a
deployable webapp.
A workaround could be to have the jobs.xml somehow redefined as a
web.xml variable. Might this be possible?

Thanks for your support.
M.

[1]

  admin:write-log('!')
  admin:write-log("1=1")


[2]
17:39:05.1000:0:0:0:0:0:0:1:42740   admin   302 745.90  [GET]
http://localhost:8984/dba
17:39:02.715SERVER  admin   INFONaN !
17:39:02.715SERVER  admin   INFONaN 1=1
17:39:02.468SERVER  admin   OK  NaN HTTP Server was started
(port: 8984).
17:39:02.454SERVER  admin   OK  NaN Server was started 
(port:
1984).


[3]
17:29:04.9330:0:0:0:0:0:0:1:52332   admin   302 1497.81 
[GET]
http://localhost:8081/contract-repository/dba
17:28:43.218SERVER  admin   OK  NaN Server was started 
(port:
1984).



On 15/10/2018 16:59, Christian Grün wrote:

Hi Marco,

I tried to reproduce the problem:

1. I downloadedhttp://files.basex.org/releases/9.0.2/BaseX902.zip
2. I unzipped the archive
3. I added a jobs.xml file in the basex/data directory:


   admin:write-log('!')


4. I started basexhttp
5. As expected, the log entry is listed in the DBA logs panel.
6. jobs:services() returns admin:write-log('!')

Could you do the same and report back to us what you did differently?

Thanks in advance
Christian





Hi Christian,
thank you as usual for your attention.
With standalone I mean basexhttp which is what we usually do. This time
we have requirement for running inside tomcat...
The jobs.xml file looks like:


if (db:exists('onedb')) then () else
db:create('ondedb')


but we have also tried with:


admin:write-log(current-dateTime())


which presents the exact same issues: basexhttp runs ok but
jobs:services() doesn't list them whereas tomcat doesn't run the scripts
but services are listed.
Thank you again,
Marco.

On 15/10/2018 15:25, Ch

Re: [basex-talk] Tomcat & services

2018-10-15 Thread Marco Lettere
Ok, actually on my personal PC jobs are correctly listed also with 
basexhttp (need to investigate why on my colleagues PC this doesn't work).


Back to the more important issue...
I put the following jobs.xml[1] file in the data folder and get [2] as 
first lines of log in dba after reboot.
I put the same file into the "dbpath" folder of my app running as tomcat 
webapp but the log looks like [3].


The feeling is that services are scheduled only in accordance to the 
startup of the http server which doesn't occur when running inside Tomcat.


In general we think that when working with tomcat one usually has to 
link an external dbpath in order to avoid getting his data removed when 
redeploying the webapp. The requirement to put jobs.xml inside the 
datapath somehow breaks the atomicity of a deployable webapp.
A workaround could be to have the jobs.xml somehow redefined as a 
web.xml variable. Might this be possible?


Thanks for your support.
M.

[1]

  admin:write-log('!')
  admin:write-log("1=1")


[2]
17:39:05.100 	0:0:0:0:0:0:0:1:42740 	admin 	302 	745.90 	[GET] 
http://localhost:8984/dba

17:39:02.715SERVER  admin   INFONaN !
17:39:02.715SERVER  admin   INFONaN 1=1
17:39:02.468SERVER  admin   OK  NaN HTTP Server was started (port: 
8984).
17:39:02.454SERVER  admin   OK  NaN Server was started (port: 1984).


[3]
17:29:04.933 	0:0:0:0:0:0:0:1:52332 	admin 	302 	1497.81 	[GET] 
http://localhost:8081/contract-repository/dba

17:28:43.218SERVER  admin   OK  NaN Server was started (port: 1984).



On 15/10/2018 16:59, Christian Grün wrote:

Hi Marco,

I tried to reproduce the problem:

1. I downloaded http://files.basex.org/releases/9.0.2/BaseX902.zip
2. I unzipped the archive
3. I added a jobs.xml file in the basex/data directory:


   admin:write-log('!')


4. I started basexhttp
5. As expected, the log entry is listed in the DBA logs panel.
6. jobs:services() returns admin:write-log('!')

Could you do the same and report back to us what you did differently?

Thanks in advance
Christian





Hi Christian,
thank you as usual for your attention.
With standalone I mean basexhttp which is what we usually do. This time
we have requirement for running inside tomcat...
The jobs.xml file looks like:


if (db:exists('onedb')) then () else
db:create('ondedb')


but we have also tried with:


admin:write-log(current-dateTime())


which presents the exact same issues: basexhttp runs ok but
jobs:services() doesn't list them whereas tomcat doesn't run the scripts
but services are listed.
Thank you again,
Marco.

On 15/10/2018 15:25, Christian Grün wrote:

Hi Marco,

I see your question didn’t get an answer yet. Maybe we can first try
to find out what Jetty does (or does not):


When running standalone with Jetty, the scripts are executed even if, in
this case, we are getting empty results when running jobs:services() in
the DBA after startup.

Some questions back:
1. By 'standalone', do you mean that you are running 'basexhttp', or
do you run Jetty and deploy BaseX as WAR file?
2. How does the jobs.xml file in the database directory look like?

Cheers,
Christian






Re: [basex-talk] Tomcat & services

2018-10-15 Thread Marco Lettere

Hi Christian,
thank you as usual for your attention.
With standalone I mean basexhttp which is what we usually do. This time 
we have requirement for running inside tomcat...

The jobs.xml file looks like:


if (db:exists('onedb')) then () else 
db:create('ondedb')



but we have also tried with:


admin:write-log(current-dateTime())


which presents the exact same issues: basexhttp runs ok but 
jobs:services() doesn't list them whereas tomcat doesn't run the scripts 
but services are listed.

Thank you again,
Marco.

On 15/10/2018 15:25, Christian Grün wrote:

Hi Marco,

I see your question didn’t get an answer yet. Maybe we can first try
to find out what Jetty does (or does not):


When running standalone with Jetty, the scripts are executed even if, in
this case, we are getting empty results when running jobs:services() in
the DBA after startup.

Some questions back:
1. By 'standalone', do you mean that you are running 'basexhttp', or
do you run Jetty and deploy BaseX as WAR file?
2. How does the jobs.xml file in the database directory look like?

Cheers,
Christian





[basex-talk] Tomcat & services

2018-10-10 Thread Marco Lettere

Dear all,

has anyone experience with exploiting the persistent jobs (services) 
feature under Tomcat?


We have installed services with the proper jobs.xml file. When running 
jobs:services() in the DBA after startup, we get the services listed.


Nevertheless none of the XQuery scripts inside the jobs is executed at 
startup of the Tomcat.


When running standalone with Jetty, the scripts are executed even if, in 
this case, we are getting empty results when running jobs:services() in 
the DBA after startup.


Any clue on what we should double-check?

Thanks,

Marco.



Re: [basex-talk] Output `out:format` function question

2018-10-08 Thread Marco Lettere

On 08/10/2018 02:39, Bridger Dyson-Smith wrote:

out:format("%06d", 256, 367) -> 000256 (: the second item() is ignored :)


Hi Dyson,

as far as I understand regarding this example the number of handled 
items depends on the format string.


out:format("%06d %06d", 256, 367)

would probably output what you expect.

Regards,

Marco.



Re: [basex-talk] BaseX GUI repo install shortcut

2018-09-25 Thread Marco Lettere

F6 + F12 sounds good enough!
Thanks for the hint.

On 25/09/2018 09:41, Christian Grün wrote:

Hi Marco,


Maybe one could extend realtime execution to the command input bar

I guess that could be fairly dangerous because a command would always
be executed whenever you type in a single character.


add a shortcut for re-executing the command bar content like ctrl-enter
for the editor area?

If you enable the "Realtime mouse focus" in the GUI Preferences, it’ll
be sufficient to move the mouse cursor over the command field and
press enter. Another option is the F6 shortcut, which focuses the
command bar (with F12, you can jump back to the editor panel). Maybe
this helps?

Christian





Regards,
Marco.

On 25/09/2018 09:25, Christian Grün wrote:

Hi Marco,

Note that you can directly edit the files in your repository. In this
case, there will be no need at all to use the REPO commands.

If you prefer to edit the files outside the repository, you could type
the REPO INSTALL command into the input bar on top of the GUI and
execute it every time after you’ve updated your files.

Hope this helps,
Christian


On Tue, Sep 4, 2018 at 12:47 PM Marco Lettere  wrote:

Hi Christian et al,

I'm not aware of any shortcut (keyboard, flag or any) for speeding up
the reinstallation of a REPO module once the XQ file is being updated
when working in the GUI.

At the moment I always reinstall from the preference menu but when
developing in continuous it's a bit tedious and often I forget it and
have to switch back from my test environment to reinstall.

Is there any way or could you think of any good shortcut to (at least
partially) automate the reinstall procedure after updating the .xqm
module in a GUI tab?

Thanks,

Marco.





Re: [basex-talk] BaseX GUI repo install shortcut

2018-09-25 Thread Marco Lettere

Ok, thanks Christian.
I'll go with the second option because we use to develop on the git repo 
and then transfer the code into a deployment area which contains the repo.
Maybe one could extend realtime execution to the command input bar or 
add a shortcut for re-executing the command bar content like ctrl-enter 
for the editor area?

Regards,
Marco.

On 25/09/2018 09:25, Christian Grün wrote:

Hi Marco,

Note that you can directly edit the files in your repository. In this
case, there will be no need at all to use the REPO commands.

If you prefer to edit the files outside the repository, you could type
the REPO INSTALL command into the input bar on top of the GUI and
execute it every time after you’ve updated your files.

Hope this helps,
Christian


On Tue, Sep 4, 2018 at 12:47 PM Marco Lettere  wrote:

Hi Christian et al,

I'm not aware of any shortcut (keyboard, flag or any) for speeding up
the reinstallation of a REPO module once the XQ file is being updated
when working in the GUI.

At the moment I always reinstall from the preference menu but when
developing in continuous it's a bit tedious and often I forget it and
have to switch back from my test environment to reinstall.

Is there any way or could you think of any good shortcut to (at least
partially) automate the reinstall procedure after updating the .xqm
module in a GUI tab?

Thanks,

Marco.





[basex-talk] Tomcat and UserPrincipal

2018-09-20 Thread Marco Lettere

Hi all,

when deploying a BaseX RestXQ module as a Tomcat webapp in a container 
that uses embedded realm based authentication is there any way to 
retrieve the login name after being redirected to the protected page?


I remember back in J2EE times there was a dedicated method 
getUserPrincipal. Is there any equivalent using "request:"?


Thanks as usual for any kind support!

Marco.



Re: [basex-talk] web:response-header & Set-Cookie

2018-09-18 Thread Marco Lettere

Hi Bogdan,
I assume the last map is for options only. All headers should go into 
the second map.

Try with:

return (

    web:response-header(

  map { 'media-type': web:content-type($path) },

  map { 'Cache-Control': 'max-age=3600,public', 'Set-Cookie': 
'JSESSIONID=' || session:id() || ';max-age=1800' },


  map { 'status' : 200}

    ),

    file:read-binary($path)

}



On 18/09/2018 00:30, Bogdan Bogucki wrote:


Hello,

When I try invoke following code:

  return (

    web:response-header(

  map { 'media-type': web:content-type($path) },

  map { 'Cache-Control': 'max-age=3600,public' },

  map { 'Set-Cookie': 'JSESSIONID=' || session:id() || 
';max-age=1800' }


    ),

    file:read-binary($path)

}

I receive error:

[XPTY0004] Unknown option 'Set-Cookie'.

I am able to set cookie using .

Following code is working:

  return (

    

  

    

    

    

    


  

    ,

    file:read-binary($path)

)

I think function web:response-header should allow set cookie.

Regards
Bogdan Bogucki





[basex-talk] BaseX GUI repo install shortcut

2018-09-04 Thread Marco Lettere

Hi Christian et al,

I'm not aware of any shortcut (keyboard, flag or any) for speeding up 
the reinstallation of a REPO module once the XQ file is being updated 
when working in the GUI.


At the moment I always reinstall from the preference menu but when 
developing in continuous it's a bit tedious and often I forget it and 
have to switch back from my test environment to reinstall.


Is there any way or could you think of any good shortcut to (at least 
partially) automate the reinstall procedure after updating the .xqm 
module in a GUI tab?


Thanks,

Marco.



Re: [basex-talk] [XQUERY] Filter and paging with window clause

2018-08-28 Thread Marco Lettere

Ok, I see. Thanks a lot for the great help again!
M.

On 28/08/2018 16:05, Christian Grün wrote:

You can use the fn:sort function for that:

   for tumbling window $w in sort(
 db:open("adatabase")[every $f in $filters satisfies $f(.)],
 (),
 function($doc) { db:path($doc) }
   )
   ...

Sorting is a blocking operator, though: All documents will need to be
checked first in order to tell which one will be the smallest or
largest hit.

Best,
Christian



On Tue, Aug 28, 2018 at 4:01 PM Marco Lettere  wrote:

 and what if, a pervert I know, would like to have the docs also
globally ordered by any sort criteria?
Any chance of squeezing the order by into this elegant FLOWR?
Thanks again.
M.

On 28/08/2018 15:41, Christian Grün wrote:

Hi Marco,

Here is a minimized representation of your last query:

for tumbling window $w in
  db:open("adatabase")[every $f in $filters satisfies $f(.)]
start $first at $s when ($s = 1 + ($page - 1) * $count)
end $last at $e when $e - $s = $count - 1
return $w

Cheers,
Christian


On Tue, Aug 28, 2018 at 3:27 PM Marco Lettere  wrote:

Oh, yes. Thank you Martin.
That is obviously a good point.
The input is a database with several thousands of documents stored in it.
The filters are a dynamic sequence of xpath expressions to be applied on the 
documents expressed as function references.
I need to check whether I can compress all this into a simple predicate as 
you've shown in your example.
where clause would have been much more easier in conjunction with the map 
operator (!) but if it works I'm going to coope with the predicate.
Thanks for your suggestion in the meanwhile.

Here a simplified example of my current approach where you could think of 
$filters as a sequence of boolean functions to be applied to the documents in 
the db.

let $seq :=
  for $doc in db:open("adatabase")
  where not(false() = ($filters ! (.($doc)))
  return $doc
let $out :=
  for tumbling window $w in $seq
  start $first at $s when ($s = 1 + ($page - 1) * $count)
  end $last at $e when $e - $s = $count - 1
  return $w

Regards,
Marco.

On 28/08/2018 14:19, Martin Honnen wrote:

On 28.08.2018 11:04, Marco Lettere wrote:

here's a question related to XQuery, sorry for being slightly off-topic.

I'm struggling to find a way to combine the windowing clause and FLOWR in order 
to get a paged result of a subset of items which respect a given filter.

Of course I'm able to get this by first applying the filter to the whole input 
and then a second FLOWR for the windowing clause.


So what is the filter condition? What is the input sequence?

The closer I get is [1] which is not what I'd need because I get 2,4,6,8,10 as 
result for page 1 but I'd really want 10 results per page thus 
2,4,6,8,10,12,14,16,18,20.

Thanks for any help on this in the meanwhile I'll stick to my naive solution.

Marco.

[1]

let $page := 1
let $count := 10
return
 for tumbling window $w in (1 to 100)
 start $first at $s when ($s = 1 + ($page - 1) * $count)
 end $last at $e when ($e - $s = $count - 1)
 return
   $w ! (if (. mod 2 = 0 ) then . else ())



Can't you simply use the filter as a predicate on the expression you use in the 
windowing for clause?

Does

let $page-size := 10
for tumbling window $page in (1 to 100)[. mod 2 = 0]
start at $sp when $sp mod $page-size = 1
end at $np when ($np + 1) mod $page-size = 1
return {$page}

help?

It returns

2 4 6 8 10 12 14 16 18 20
22 24 26 28 30 32 34 36 38 40
42 44 46 48 50 52 54 56 58 60
62 64 66 68 70 72 74 76 78 80
82 84 86 88 90 92 94 96 98 100








Re: [basex-talk] [XQUERY] Filter and paging with window clause

2018-08-28 Thread Marco Lettere
 and what if, a pervert I know, would like to have the docs also 
globally ordered by any sort criteria?

Any chance of squeezing the order by into this elegant FLOWR?
Thanks again.
M.

On 28/08/2018 15:41, Christian Grün wrote:

Hi Marco,

Here is a minimized representation of your last query:

   for tumbling window $w in
 db:open("adatabase")[every $f in $filters satisfies $f(.)]
   start $first at $s when ($s = 1 + ($page - 1) * $count)
   end $last at $e when $e - $s = $count - 1
   return $w

Cheers,
Christian


On Tue, Aug 28, 2018 at 3:27 PM Marco Lettere  wrote:

Oh, yes. Thank you Martin.
That is obviously a good point.
The input is a database with several thousands of documents stored in it.
The filters are a dynamic sequence of xpath expressions to be applied on the 
documents expressed as function references.
I need to check whether I can compress all this into a simple predicate as 
you've shown in your example.
where clause would have been much more easier in conjunction with the map 
operator (!) but if it works I'm going to coope with the predicate.
Thanks for your suggestion in the meanwhile.

Here a simplified example of my current approach where you could think of 
$filters as a sequence of boolean functions to be applied to the documents in 
the db.

   let $seq :=
 for $doc in db:open("adatabase")
 where not(false() = ($filters ! (.($doc)))
 return $doc
   let $out :=
 for tumbling window $w in $seq
 start $first at $s when ($s = 1 + ($page - 1) * $count)
 end $last at $e when $e - $s = $count - 1
 return $w

Regards,
Marco.

On 28/08/2018 14:19, Martin Honnen wrote:

On 28.08.2018 11:04, Marco Lettere wrote:

here's a question related to XQuery, sorry for being slightly off-topic.

I'm struggling to find a way to combine the windowing clause and FLOWR in order 
to get a paged result of a subset of items which respect a given filter.

Of course I'm able to get this by first applying the filter to the whole input 
and then a second FLOWR for the windowing clause.


So what is the filter condition? What is the input sequence?

The closer I get is [1] which is not what I'd need because I get 2,4,6,8,10 as 
result for page 1 but I'd really want 10 results per page thus 
2,4,6,8,10,12,14,16,18,20.

Thanks for any help on this in the meanwhile I'll stick to my naive solution.

Marco.

[1]

let $page := 1
let $count := 10
return
for tumbling window $w in (1 to 100)
start $first at $s when ($s = 1 + ($page - 1) * $count)
end $last at $e when ($e - $s = $count - 1)
return
  $w ! (if (. mod 2 = 0 ) then . else ())



Can't you simply use the filter as a predicate on the expression you use in the 
windowing for clause?

Does

let $page-size := 10
for tumbling window $page in (1 to 100)[. mod 2 = 0]
   start at $sp when $sp mod $page-size = 1
   end at $np when ($np + 1) mod $page-size = 1
return {$page}

help?

It returns

2 4 6 8 10 12 14 16 18 20
22 24 26 28 30 32 34 36 38 40
42 44 46 48 50 52 54 56 58 60
62 64 66 68 70 72 74 76 78 80
82 84 86 88 90 92 94 96 98 100








Re: [basex-talk] [XQUERY] Filter and paging with window clause

2018-08-28 Thread Marco Lettere

Wow!
Works ... of course :-D
And I'd say rather fast!
Thanks a lot Christian and Martin!
Auf bald!

On 28/08/2018 15:41, Christian Grün wrote:

Hi Marco,

Here is a minimized representation of your last query:

   for tumbling window $w in
 db:open("adatabase")[every $f in $filters satisfies $f(.)]
   start $first at $s when ($s = 1 + ($page - 1) * $count)
   end $last at $e when $e - $s = $count - 1
   return $w

Cheers,
Christian


On Tue, Aug 28, 2018 at 3:27 PM Marco Lettere  wrote:

Oh, yes. Thank you Martin.
That is obviously a good point.
The input is a database with several thousands of documents stored in it.
The filters are a dynamic sequence of xpath expressions to be applied on the 
documents expressed as function references.
I need to check whether I can compress all this into a simple predicate as 
you've shown in your example.
where clause would have been much more easier in conjunction with the map 
operator (!) but if it works I'm going to coope with the predicate.
Thanks for your suggestion in the meanwhile.

Here a simplified example of my current approach where you could think of 
$filters as a sequence of boolean functions to be applied to the documents in 
the db.

   let $seq :=
 for $doc in db:open("adatabase")
 where not(false() = ($filters ! (.($doc)))
 return $doc
   let $out :=
 for tumbling window $w in $seq
 start $first at $s when ($s = 1 + ($page - 1) * $count)
 end $last at $e when $e - $s = $count - 1
 return $w

Regards,
Marco.

On 28/08/2018 14:19, Martin Honnen wrote:

On 28.08.2018 11:04, Marco Lettere wrote:

here's a question related to XQuery, sorry for being slightly off-topic.

I'm struggling to find a way to combine the windowing clause and FLOWR in order 
to get a paged result of a subset of items which respect a given filter.

Of course I'm able to get this by first applying the filter to the whole input 
and then a second FLOWR for the windowing clause.


So what is the filter condition? What is the input sequence?

The closer I get is [1] which is not what I'd need because I get 2,4,6,8,10 as 
result for page 1 but I'd really want 10 results per page thus 
2,4,6,8,10,12,14,16,18,20.

Thanks for any help on this in the meanwhile I'll stick to my naive solution.

Marco.

[1]

let $page := 1
let $count := 10
return
for tumbling window $w in (1 to 100)
start $first at $s when ($s = 1 + ($page - 1) * $count)
end $last at $e when ($e - $s = $count - 1)
return
  $w ! (if (. mod 2 = 0 ) then . else ())



Can't you simply use the filter as a predicate on the expression you use in the 
windowing for clause?

Does

let $page-size := 10
for tumbling window $page in (1 to 100)[. mod 2 = 0]
   start at $sp when $sp mod $page-size = 1
   end at $np when ($np + 1) mod $page-size = 1
return {$page}

help?

It returns

2 4 6 8 10 12 14 16 18 20
22 24 26 28 30 32 34 36 38 40
42 44 46 48 50 52 54 56 58 60
62 64 66 68 70 72 74 76 78 80
82 84 86 88 90 92 94 96 98 100








Re: [basex-talk] [XQUERY] Filter and paging with window clause

2018-08-28 Thread Marco Lettere

Oh, yes. Thank you Martin.
That is obviously a good point.
The input is a database with several thousands of documents stored in it.
The filters are a dynamic sequence of xpath expressions to be applied on 
the documents expressed as function references.
I need to check whether I can compress all this into a simple predicate 
as you've shown in your example.
where clause would have been much more easier in conjunction with the 
map operator (!) but if it works I'm going to coope with the predicate.

Thanks for your suggestion in the meanwhile.

Here a simplified example of my current approach where you could think 
of /$filters/ as a sequence of boolean functions to be applied to the 
documents in the db.


  let $seq :=
    for $doc in db:open("adatabase")
    where not(false() = ($filters ! (.($doc)))
    return $doc
  let $out :=
    for tumbling window $w in $seq
    start $first at $s when ($s = 1 + ($page - 1) * $count)
    end $last at $e when $e - $s = $count - 1
    return $w

Regards,
Marco.

On 28/08/2018 14:19, Martin Honnen wrote:

On 28.08.2018 11:04, Marco Lettere wrote:


here's a question related to XQuery, sorry for being slightly off-topic.

I'm struggling to find a way to combine the windowing clause and 
FLOWR in order to get a paged result of a subset of items which 
respect a given filter.


Of course I'm able to get this by first applying the filter to the 
whole input and then a second FLOWR for the windowing clause.


So what is the filter condition? What is the input sequence?

The closer I get is [1] which is not what I'd need because I get 
2,4,6,8,10 as result for page 1 but I'd really want 10 results per 
page thus 2,4,6,8,10,12,14,16,18,20.


Thanks for any help on this in the meanwhile I'll stick to my naive 
solution.


Marco.

[1]

let $page := 1
let $count := 10
return
   for tumbling window $w in (1 to 100)
   start $first at $s when ($s = 1 + ($page - 1) * $count)
   end $last at $e when ($e - $s = $count - 1)
   return
 $w ! (if (. mod 2 = 0 ) then . else ())



Can't you simply use the filter as a predicate on the expression you 
use in the windowing for clause?


Does

let $page-size := 10
for tumbling window $page in (1 to 100)[. mod 2 = 0]
  start at $sp when $sp mod $page-size = 1
  end at $np when ($np + 1) mod $page-size = 1
return {$page}

help?

It returns

2 4 6 8 10 12 14 16 18 20
22 24 26 28 30 32 34 36 38 40
42 44 46 48 50 52 54 56 58 60
62 64 66 68 70 72 74 76 78 80
82 84 86 88 90 92 94 96 98 100







[basex-talk] [XQUERY] Filter and paging with window clause

2018-08-28 Thread Marco Lettere

Hi all,

here's a question related to XQuery, sorry for being slightly off-topic.

I'm struggling to find a way to combine the windowing clause and FLOWR 
in order to get a paged result of a subset of items which respect a 
given filter.


Of course I'm able to get this by first applying the filter to the whole 
input and then a second FLOWR for the windowing clause.


But I was wondering whether it's possible to streamline this into one 
single FLOWR.


Two bonus questions:

1. Will this actually be more efficient than the naive (two FLOWR)
   solution or is the almighty optimizer able to blend the two
   subsequent FLOWR on its own?
2. What if the input is unordered and I also would like to introduce
   sorting?

The closer I get is [1] which is not what I'd need because I get 
2,4,6,8,10 as result for page 1 but I'd really want 10 results per page 
thus 2,4,6,8,10,12,14,16,18,20.


Thanks for any help on this in the meanwhile I'll stick to my naive 
solution.


Marco.

[1]

let $page := 1
let $count := 10
return
  for tumbling window $w in (1 to 100)
  start $first at $s when ($s = 1 + ($page - 1) * $count)
  end $last at $e when ($e - $s = $count - 1)
  return
    $w ! (if (. mod 2 = 0 ) then . else ())



Re: [basex-talk] IETF Date in If-Modified-Since header

2018-05-29 Thread Marco Lettere

On 29/05/2018 13:58, Christian Grün wrote:

Hi Marco,

The current RESTXQ 1.0 specification seems to mandate that comma 
separated lists needs to be extracted to single items [1]. There isn’t 
that much activity on the EXQuery repository [2], but maybe it could 
be discussed over there?

Yes. Going to ask there.



As an alternative, you can use the request:header function to retrieve 
the full value [3].



Much better solution than mine! I'll follow this advice. Thanks!


Best,
Christian

[1] 
http://exquery.github.io/exquery/exquery-restxq-specification/restxq-1.0-specification.html#form-header-annotation

[2] https://github.com/exquery/exquery/issues
[3] http://docs.basex.org/wiki/Request_Module#request:header



On Tue, May 29, 2018 at 1:49 PM, Marco Lettere <mailto:m.lett...@gmail.com>> wrote:


Hello all,

need to use the If-Modified-Since header in one of my APIs. The
format requested is the IETF date format:

If-Modified-Since: Wed, 21 Oct 2015 07:28:00 GMT

as explained in [1].

If I pass the header as prescribed to a restxq made like:

declare
  %rest:path("config")
  %rest:header-param("If-Modified-Since", "{$modifiedsince-ietf}")
  %rest:HEAD
function config:check($modifiedsince-ietf as xs:string?){

I get the following exception:

Cannot convert xs:string+ to xs:string?: ("Wed", "21 Oct 2015
07:28:00 GMT").

Which somehow demonstrates that , is treated as a separator of
values for the header.

I patched by replacing ? with * and using a string-join on the
$modifiedsince-ietf before passing it to the follwoing
parse-ietf-date function. But this is hard to explain and probably
subject to other issues in the future.

Is there a way to pass HTTP headers verbatim to the RestXQ function?

thanks!

M.


[1]
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since

<https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since>






[basex-talk] IETF Date in If-Modified-Since header

2018-05-29 Thread Marco Lettere

Hello all,

need to use the If-Modified-Since header in one of my APIs. The format 
requested is the IETF date format:


If-Modified-Since: Wed, 21 Oct 2015 07:28:00 GMT

as explained in [1].

If I pass the header as prescribed to a restxq made like:

declare
  %rest:path("config")
  %rest:header-param("If-Modified-Since", "{$modifiedsince-ietf}")
  %rest:HEAD
function config:check($modifiedsince-ietf as xs:string?){

I get the following exception:

Cannot convert xs:string+ to xs:string?: ("Wed", "21 Oct 2015 07:28:00 
GMT").


Which somehow demonstrates that , is treated as a separator of values 
for the header.


I patched by replacing ? with * and using a string-join on the 
$modifiedsince-ietf before passing it to the follwoing parse-ietf-date 
function. But this is hard to explain and probably subject to other 
issues in the future.


Is there a way to pass HTTP headers verbatim to the RestXQ function?

thanks!

M.


[1] 
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since




[basex-talk] Documentation inquiries

2018-05-16 Thread Marco Lettere

Hi all,

investigating the new features tagged 9.x, I discovered a lot of great 
stuff. I have teo smaller questions ...


I'd like to know whether the new "-c flag" for the Basex Http Server in 
[1] means that one can request a script of commands to be executed once 
at startup of the server. Is there a way to do something equivalent also 
at server shutdown?


In [2] with the new Service it is said that one could write jobs.xml 
file inside the database directory. I couldn't find the specification of 
the syntax of this file anywhere. Do you have some hints?


Bye,

Marco.

[1] http://docs.basex.org/wiki/Command-Line_Options#HTTP_Server

[2] http://docs.basex.org/wiki/Jobs_Module#Services



Re: [basex-talk] baseX vs ExistDB

2018-04-20 Thread Marco Lettere

On 20/04/2018 15:49, Christian Grün wrote:

Hi Omar,

Thank you (and everyone else) for sharing your experiences.


* BaseX in comparison to exist-db turned out to be particularly bad at
hosting multiple XQuery based applications like RestXQ endpoint in one
instance.

Definitely true; BaseX was not built for that. If you want to run
multiple applications with a single web server, the recommended
approach is to use the WAR distributions of BaseX and deploy each
application as a separate servlet.

Or use it as we do here. A new BaseX process for every application. A 
sort of very small application container or nano-service with the 
optimization that they all share the BaseX code and just use different 
restxq and possibly data folders.


;-)

M.



Re: [basex-talk] Validation Module: validate:xsd-report( ) improvement

2018-04-19 Thread Marco Lettere

Hi Yitzhak,


maybe, by slightly rewriting your code,  you could remove the unwnated 
attribute and serialize your output to a file and view it from a text 
editor?



let $xml := 
'd:\Temp\CDW\HOME\id4879_BO201801_HomeSubscriberMovementFact.xml'


let $xsd := 'd:\Temp\CDW\HOME\HomeSubscriberMovementFact.xsd'

let $validate := validate:xsd-report($xml, $xsd, '1.1')

return file:write("output.xml",
  copy $newvalidate := $validate
  modify (delete node $newvalidate//@url)
  return $newvalidate
)


Regards,

Marco.


return validate:xsd-report($xml, $xsd, '1.1')

On 19/04/2018 19:20, Yitzhak Khabinsky wrote:


Hello,

I am successfully using BaseX Validation Module.

Along the following lines:

let $xml := 
'd:\Temp\CDW\HOME\id4879_BO201801_HomeSubscriberMovementFact.xml'


let $xsd := 'd:\Temp\CDW\HOME\HomeSubscriberMovementFact.xsd'

return validate:xsd-report($xml, $xsd, '1.1')

My XML files have multi-megabyte size and lots of validation errors. 
In tens or hundreds of thousands of errors.


Behind the scenes, Saxon validator 9.8.0.11 is running.

Unfortunately, the output structure contains a repeating *url* attribute.

The BaseX output pane cannot present all the errors.

It says: “(Chopped) Results”.


invalid
level="Error"line="10"column="26"url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The 
content "N/A" of element CommercialServiceCode does not match 
the required simple type. Value "N/A" contravenes the enumeration 
facet "R60080-X00162, R60080-X00163, ..." of the type 
Q{http://www.millicom.com}CommercialServiceCodeType
level="Error"line="19"column="23"url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The 
content "TBD" of element MovementTechnology does not match the 
required simple type. Value "TBD" contravenes the enumeration facet 
"N/A, HFC, GPON, MMDS, FIBER, C..." of the type 
Q{http://www.millicom.com}MovementTechnologyType
level="Error"line="24"column="18"url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The 
content "-1." of element DownloadSpeed does not match the 
required simple type. Value "-1" contravenes the minExclusive facet 
"0" of the type Q{http://www.millicom.com}DownloadSpeedType
level="Error"line="26"column="6"url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The 
7th field in constraint {PK} has no value

...


My proposal is to eliminate the repeated *url* attribute from the each 
*message* and elevate it to its own element just once under the root 
*report* tag.


Along the following output structure:



invalid
file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml
The content "N/A" of 
element CommercialServiceCode does not match the required 
simple type. Value "N/A" contravenes the enumeration facet 
"R60080-X00162, R60080-X00163, ..." of the type 
Q{http://www.millicom.com}CommercialServiceCodeType
The content "TBD" of 
element MovementTechnology does not match the required simple 
type. Value "TBD" contravenes the enumeration facet "N/A, HFC, GPON, 
MMDS, FIBER, C..." of the type 
Q{http://www.millicom.com}MovementTechnologyType
The content "-1." of 
element DownloadSpeed does not match the required simple type. 
Value "-1" contravenes the minExclusive facet "0" of the type 
Q{http://www.millicom.com}DownloadSpeedType
The 7th field in constraint 
{PK} has no value

...


This way the output of the validation is much more readable and 
hopefully will fit in its entirety to the output pane.


Regards,

Yitzhak Khabinsky

Technical Services Lead

Millicom International Services LLC

396 Alhambra Circle, Suite 1100

Coral Gables, FL  33134

Skype4B: +1 (305) 445-4172

Tel: (954) 684-8673

_yitzhak.khabin...@millicom.com _

www.millicom.com 







Re: [basex-talk] Multipart and charset

2018-04-16 Thread Marco Lettere

Great,
this allows us to remove a lot of proprietary and risky XQuery code to 
circumvent the issue.
And at the moment I also think that supporting more byzantine scenarios 
is less important.

Thanks a lot.
Marco.

On 16/04/2018 10:37, Christian Grün wrote:

Hi Marco,

I didn’t forget your mail ;)

Thanks for the elaborate analysis, which clearly helped to me to fix
the issue more quickly [1]: Quoted strings will now be detected, and
backslashed characters within the value will be unescaped. I didn’t
add support for comments so far (they seem to be pretty rare these
days), but I might tackle this if someone else encounters problems.

A new snapshot is available [2]. BaseX 9.0.1 will be due next week.

Cheers,
Christian

[1] 
https://github.com/BaseXdb/basex/commit/eae493e0ba2e6d5989f2fe8734ede6df380ac54e
[2] http://files.basex.org/releases/latest/




On Thu, Apr 5, 2018 at 1:20 PM, Marco Lettere <m.lett...@gmail.com> wrote:

Hi everyone,

when parsing a multipart response we are getting from a server we need to
integrate with basex raises the exception [1].

By forcing the mediatype (with override-media-type attribute) to text/plain
we've been able to see that parts look something like [2].

Our suspect was that the multipart parser does not tolerate the quotes
around UTF-8 but exploring the jungles of RFC they seem to be allowed as per
an example of https://tools.ietf.org/html/rfc2045 that states:

Note that the value of a quoted string parameter does not include the
quotes. That is, the quotation marks in a quoted-string are not a part of
the value of the parameter, but are merely used to delimit that parameter
value. In addition, comments are allowed in accordance with RFC 822 rules
for structured header fields.
Thus the following two forms

Content-type: text/plain; charset=us-ascii (Plain text)

Content-type: text/plain; charset="us-ascii"

are completely equivalent.

Could we kindly have some feedback on this in order to avoid having to parse
the whole multipart on our own?

Thank you very much.

Marco.

[1]

Unsupported encoding: java.nio.charset.IllegalCharsetNameException: "UTF-8"
at org.basex.core.Command.execute(Command.java:94)
at org.basex.gui.GUI.exec(GUI.java:427)
at org.basex.gui.GUI.lambda$4(GUI.java:370)
at org.basex.gui.GUI$$Lambda$80/502071500.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.basex.query.QueryException: Unsupported encoding:
java.nio.charset.IllegalCharsetNameException: "UTF-8"
at org.basex.query.QueryError.get(QueryError.java:1392)
[...]

[2]
boundary44723329.705882352941176--
Content-Type: application/xop+xml; type="application/soap+xml";
charset="UTF-8"
Content-Transfer-Encoding: 8bit
Content-Id: <0.1808ACE4.38B8.11E8.8E46.50569966F200>





[basex-talk] Multipart and charset

2018-04-05 Thread Marco Lettere

Hi everyone,

when parsing a multipart response we are getting from a server we need 
to integrate with basex raises the exception [1].


By forcing the mediatype (with override-media-type attribute) to 
text/plain we've been able to see that parts look something like [2].


Our suspect was that the multipart parser does not tolerate the quotes 
around UTF-8 but exploring the jungles of RFC they seem to be allowed as 
per an example of https://tools.ietf.org/html/rfc2045 that states:


Note that the value of a quoted string parameter does not include the 
quotes. That is, the quotation marks in a quoted-string are not a part 
of the value of the parameter, but are merely used to delimit that 
parameter value. In addition, comments are allowed in accordance with 
RFC 822 rules for structured header fields.

Thus the following two forms

Content-type: text/plain; charset=us-ascii (Plain text)

Content-type: text/plain; charset="us-ascii"

are completely equivalent.

Could we kindly have some feedback on this in order to avoid having to 
parse the whole multipart on our own?


Thank you very much.

Marco.

[1]

Unsupported encoding: java.nio.charset.IllegalCharsetNameException: "UTF-8"
at org.basex.core.Command.execute(Command.java:94)
at org.basex.gui.GUI.exec(GUI.java:427)
at org.basex.gui.GUI.lambda$4(GUI.java:370)
at org.basex.gui.GUI$$Lambda$80/502071500.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745) Caused by: 
org.basex.query.QueryException: Unsupported encoding: 
java.nio.charset.IllegalCharsetNameException: "UTF-8"

at org.basex.query.QueryError.get(QueryError.java:1392)
[...]

[2]
boundary44723329.705882352941176--
Content-Type: application/xop+xml; type="application/soap+xml"; 
charset="UTF-8"

Content-Transfer-Encoding: 8bit
Content-Id: <0.1808ACE4.38B8.11E8.8E46.50569966F200>


Re: [basex-talk] BaseX 9.0: The Spring Edition!

2018-03-26 Thread Marco Lettere

Thank you all very much for this great tool!
I'm eagerly looking forward to use the new version and the new site 
layout looks amazing.

Well done!

M.

On 23/03/2018 21:32, Christian Grün wrote:

Dear BaseX aficionados,

We are very happy to announce the release of BaseX 9.0!

The new version of our XML database system and XQuery 3.1 processor
includes some great new features and a vast number of minor
improvements and optimizations. It’s both the usage of BaseX in
productive environments as well as the valuable feedback of our open
source users that make BaseX better and better, and that allow and
motivate us to keep going. Thanks to all of you!

Along with the new release, we invite you to visit our relaunched
homepage: http://basex.org/.

Java 8 is now required to run BaseX. The most prominent features of
Version 9.0 are:

  XQUERY
  - Comprehensive extensions in the internal XQuery optimizer framework
  - Dynamic hash joins rewritings of general comparisons
  - Register query jobs as persistent services
  - Process very large CSV files via the new CSV 'xquery' format
  - Update Module: higher-order functions for performing updates
  - Unified static typing, including maps, arrays and function items
  - COPYNODE: Lightweight copying of XML fragments
  - ENFORCEINDEX: Enforce rewriting for index access

  STORAGE
  - Improved database compression (short strings, whitespaces)

  WEB APPLICATIONS
  - Permissions layer: Unified definition of RESTXQ access patterns
  - RESTXQ: Full support for client- and server-side quality factors
  - REST: Run BaseX command scripts
  - Web server upgrade to Jetty 9

  REPOSITORIES
  - Combined packaging mechanism (XQuery and Java)
  - Java Packaging: Bundling of additional libraries

  DISTRIBUTIONS
  - Revised detection and configuration of BaseX home directory
  - Windows executable: better detection of Java libraries

  DBA: BASEX DATABASE ADMINISTRATION
  - Improved editing and evaluation of XQuery modules
  - Revised file, session and job management

More details can be found on our homepage. We are looking forward to
your feedback.

Enjoy and have fun,
Christian, BaseX Team





[basex-talk] Uncomprehensible (at least to me) behaviour with external bindings

2018-03-21 Thread Marco Lettere

Hi all,

trying to understand why we get the exception [2] running the code [1] 
in a Java app, I stumbled upon a strange behaviour of the GUI.


I rewrote the query to:

xquery:eval("
declare variable $_dep as xs:long* external;
$_dep[2] * $_dep[2]", map{ "_dep" : (1,2,3)})

obtaining the correct result (4).

Anyway if I assign (1,2,3) to a binding through the proper dialog of the 
UI I get the following exception at execution:


Error:
Stopped at 
/home/lettere/git/dasbox-frontend/dasbox-frontend-ui/webapp/ui/resources/js/monitoring/file, 
2/12:

[FORG0001] Cannot cast to xs:double: (1,2,3).

Strangely enough this happens even if I leave the map overriding locally 
the values for the external parameters in the xquery:eval. The dialog wins?


If I rewrite the query like this and remove the parameter values from 
the dialog:


xquery:eval("
declare variable $_dep as xs:long* external := (1,2,3);

$_dep[2] * $_dep[2]")

I get yet another typecast exception:

Error:
Stopped at 
/home/lettere/git/dasbox-frontend/dasbox-frontend-ui/webapp/ui/resources/js/monitoring/file, 
2/12:

[XPTY0004] Cannot promote xs:integer+ to $_dep as xs:long*: (1, 2, 3).

Is there a nice wording to explain all this different behaviour and 
might it be that the exception we get in the Java embedded version is 
somehow related to the same?


Thanks a lot for helping,

Marco.

[1]

declare variable $_history as xs:long* external;
declare variable $_now as xs:dateTime external;
declare variable $_dep as xs:long* external;
declare variable $dep as xs:long external;

$_dep[2] * $_dep[2]

[2]

Caused by: org.basex.query.QueryException: [FORG0001] Cannot cast to 
xs:double: [Ljava.lang.Object;@2f0539f0.
    at org.basex.query.QueryError.get(QueryError.java:1365) 
~[BaseX.jar:8.6.7]
    at org.basex.query.QueryError.castError(QueryError.java:1571) 
~[BaseX.jar:8.6.7]
    at org.basex.query.value.item.Dbl.parse(Dbl.java:180) 
~[BaseX.jar:8.6.7]
    at org.basex.query.value.item.Item.dbl(Item.java:134) 
~[BaseX.jar:8.6.7]
    at org.basex.query.value.type.AtomType.checkLong(AtomType.java:927) 
~[BaseX.jar:8.6.7]
    at org.basex.query.value.type.AtomType$20.cast(AtomType.java:331) 
~[BaseX.jar:8.6.7]
    at org.basex.query.value.type.AtomType$20.cast(AtomType.java:1) 
~[BaseX.jar:8.6.7]
    at org.basex.query.QueryContext.cast(QueryContext.java:733) 
~[BaseX.jar:8.6.7]
    at org.basex.query.QueryContext.bind(QueryContext.java:486) 
~[BaseX.jar:8.6.7]
    at org.basex.query.QueryProcessor.bind(QueryProcessor.java:129) 
~[BaseX.jar:8.6.7]
    at com.nw.basex.xquery.XQueryExecutor.bind(XQueryExecutor.java:89) 
~[com.nw.dasbox-2018-03-15.jar:?]
    at 
com.nw.basex.xquery.XQueryExecutor.execute(XQueryExecutor.java:107) 
~[com.nw.dasbox-2018-03-15.jar:?]
    at 
com.nw.dasbox.vdr.resolution.InlineXQueryResolutionEngine.evaluate(InlineXQueryResolutionEngine.java:91) 
~[com.nw.dasbox-2018-03-15.jar:?]

    ... 10 more


[basex-talk] 9.0?

2018-01-29 Thread Marco Lettere

Hi Christian & team,

since XML Prague 18 is approaching I remembered that there used to be 
this wonderful habit of releasing a new BaseX version for the event :-).


I am wondering whether there are any timing plans for the 9.0 release.

In particular we are eager to test the new features [1] and [2].

Thank you for any updates on the matter and please continue with your 
incredible work!


Best regards,

Marco.

[1] https://github.com/BaseXdb/basex/issues/1498 (persistent jobs)

[2] https://github.com/BaseXdb/basex/issues/1449 (websocket support)



Re: [basex-talk] BaseX Sharding with ActiveMQ & Docker

2018-01-05 Thread Marco Lettere

Hi James,
as to my experience the approach works.
I've tried to use N Slaves BaseX (DB only) instances coordinated by one 
Master BaseX including HTTP server. On the latter I implemented a RestXQ 
[1] receiving a HTTP request which was then turned into a XQuery and 
executed on the different backend instances through the client module 
[2]. I have been able to achieve a scalability of nearly 80% on some 
particular queries.
The results were encouraging indeed but a few details related to 
parallel computing in general have to be considered.
1) If the query is memory only then you can exploit multi-cores to 
actually achieve parallelism. If the query has to heavily access DBs on 
disk (most of the cases actually are io-bound) then you should ensure 
that your distributed Slaves access different disks or reside on 
different PCs otherwise all the gain of  parallel computation is lost by 
overhead on disk access.
2) Uploading of documents from a single source usually doesn't exploit 
data-parallelism because all the data has to be sequentialized through a 
distributor node first. So if you are able to generate the data from the 
beginning in a distributed way, this could improve the distribution step 
too.
3) Another thing to keep in mind is that you should try to perform as 
much of the work on the Slaves (for instance query and transformation) 
and limit the work on the coordinator to a bare minimum of computing and 
memory usage in order to avoid the coordinator to be flooded with the 
results produced by all your slaves. I achieved this by injecting 
functions as external variables in the XQ to be executed on the Slaves.


Hope this is useful for your work and I'm really looking forward to know 
how it will proceed!

Regards,
Marco.

[1] http://docs.basex.org/wiki/RESTXQ
[2] http://docs.basex.org/wiki/Client_Module

On 03/01/2018 19:50, James Sears wrote:


In the past I hit a scalability limit with BaseX - a billion+ nodes 
kind of a made querying it a bit slower than I liked.


I thought I'd try and address this, so I’ve written some code and 
placed it in GitHub: https://github.com/jameshnsears/xqa-documentation


What I've done is proof of concept, that's all - no way "finished". 
I'm emailing the list in the hope that what I've done so far might 
generate some constructive criticism. Maybe my approach has potential, 
maybe it doesn't?


There are only four components so far, the first three are Docker 
containers:


* an ActiveMQ instance

* a load balancer

* a shard

* a command line client exists to load the XML, from file, into an 
ActiveMQ queue.


The software requires close to zero configuration. For example, each 
shard you start will automatically receive XML from the load balancer. 
And the load balancer distributes XML so that each shard holds the 
same # of documents.


There's a Travis project associated with the above link - it shows how 
easy it is to run the software end to end.


So far my effort is all about ingesting the XML, before I move further 
I thought I'd canvass some feedback - so if anyone has any then please 
give it :-)


Thanks.





Re: [basex-talk] TLS 1.2

2017-12-18 Thread Marco Lettere

Great! Thanks to you!
M.

On 18/12/2017 20:18, Christian Grün wrote:

Thanks, Marco. Your pull request has been merged, and the
documentation has been updated [1].

Cheers,
Christian

[1] http://docs.basex.org/wiki/Options#IGNOREHOSTNAME



On Mon, Dec 18, 2017 at 10:44 AM, Marco Lettere <m.lett...@gmail.com> wrote:

Hi Christian,
I've filed a pull request [1] to the github repo.
I've tested it with our use case and it works fine.
Check it out and let me know.
Regards,
Marco.

[1] https://github.com/BaseXdb/basex/pull/1533


On 17/12/2017 22:58, Christian Grün wrote:

Hi Marco,

Thanks for reporting all the details back to us.


Final question to Christian ... instead of ignoring certificates
altogether
could you plan for a less risky option IGNORE_HOSTNAME_VERIFICATION which
just ignores the verification of the host name of a certificate?

I think so. Certificates are currently ignored via a small Java
function [1]. Do you have an idea how this code would need to be
modified?

Cheers,
Christian

[1]
https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/io/IOUrl.java#L139-L161







Re: [basex-talk] TLS 1.2

2017-12-18 Thread Marco Lettere

Hi Christian,
I've filed a pull request [1] to the github repo.
I've tested it with our use case and it works fine.
Check it out and let me know.
Regards,
Marco.

[1] https://github.com/BaseXdb/basex/pull/1533

On 17/12/2017 22:58, Christian Grün wrote:

Hi Marco,

Thanks for reporting all the details back to us.


Final question to Christian ... instead of ignoring certificates altogether
could you plan for a less risky option IGNORE_HOSTNAME_VERIFICATION which
just ignores the verification of the host name of a certificate?

I think so. Certificates are currently ignored via a small Java
function [1]. Do you have an idea how this code would need to be
modified?

Cheers,
Christian

[1] 
https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/io/IOUrl.java#L139-L161





Re: [basex-talk] TLS 1.2

2017-12-13 Thread Marco Lettere

Hi all,
just a bit of a wrap up in case someone could use it in the future ...

The usecase:
Connect to an Identitiy provider through HTTPS based on TLSv1.2 with 
client authentication required (a password protected PKCS12 file 
"clientCERTKEY.p12" with client certificate and key was provided 
together with a CA certificate for trusting the server certificate 
"serverCA.crt"). Extract a chunk of XML from the response which 
represents a SAML assertion and and use it to call another service again 
over TLSv1.2 with the same client certificate/key pair.
TLSv1.2 with mutual authentication requires a longer SSL handshake 
during which both Server and Client are authenticated and trusted.


Objective:
Do as much as possible in XQuery without descending into the Java dungeon.

Solution:

1) import the serverCA.crt int a dedicated Java keystore 
(localtruststore.jks) that will act as truststore. This works probably 
also if the cert is imported in the global truststore but we did not 
want to pollute the global store while testing. Note that you have to 
set a password to protect the truststore during importing.


/keytool -v -importcert -keystore localtruststore.jks -file serverCA.crt

/2) instruct basex to use the newly created truststore plus the 
clientCERTKEY.p12 file as keystore for the client. This has been 
resolved by modifying the BaseX startup line to look like:

/
//java -Djavax.net.ssl.keyStore=path/to/clientCERTKEY.p12 
-Djavax.net.ssl.keyStorePassword=THEGIVENPASSFORP12 
-Djavax.net.ssl.keyStoreType=PKCS12 
-Djavax.net.ssl.trustStore=path/to/localtruststore.jks 
-Djavax.net.ssl.trustStorePassword=THEPASSWORDFORTRUSTSTORE -jar BaseX.jar

/
3) Up to this point it worked fine. I just used http:send-request to 
send my request from a basexgui for instance and I was easily able to 
migrate the SAML assertion from the response to the request for the 
second service.


4) The second call failed because of an annoying (but rather common) 
issue with the server certificate whose DN entries did not match 
correctly the requiremnts for the second service url. So I had to write 
a Custom HostnameVerifier in Java (plenty of examples on the web). And 
unfortunately I had to pollute my xquery workflow (put the jar in 
library folder and link it up in the code like below).


/Java code for the verifier://
//package com.nw.tls;//
//
//import javax.net.ssl.HostnameVerifier;//
//import javax.net.ssl.HttpsURLConnection;//
//import javax.net.ssl.SSLSession;//
//
//public class LaxHostNameVerifier implements HostnameVerifier{//
//
//    public LaxHostNameVerifier() {//
//HttpsURLConnection.setDefaultHostnameVerifier(this);//
//    }//

//    @Override//
//    public boolean verify(String hostname, SSLSession session) {//
//        return true//;
//    }//
//}/

Code for including it into XQuery
/declare namespace Verifier = "java:com.nw.tls.LaxHostNameVerifier";//
//let $init := Verifier:new()/

5) In a last effort we've found out a way to link extra jars by 
declaring them as java agents on the startup line. So I was finally able 
to use my pure XQuery flow by just enriching the startup line like below:


/java -Djavax.net.ssl.keyStore=path/to/clientCERTKEY.p12 
-Djavax.net.ssl.keyStorePassword=THGIVENPASSFORP12 
-Djavax.net.ssl.keyStoreType=PKCS12 
-Djavax.net.ssl.trustStore=path/to/localtruststore.jks 
-Djavax.net.ssl.trustStorePassword=THEPASSWORDFORTRUSTSTORE 
-javaagent:LaxHostnameVerifierAgent.jar -jar BaseX.jar/


Thats'all folks.
Final question to Christian ... instead of ignoring certificates 
altogether could you plan for a less risky option 
IGNORE_HOSTNAME_VERIFICATION which just ignores the verification of the 
host name of a certificate? I think this is such a common case that it 
might deserve a dedicated flag. Isn't it?


Thanks for the support.
Kind regards and Seaon Greetings to wveryone!
Marco.

On 11/12/2017 21:41, Christian Grün wrote:

That’s good news, Marco!

All I remember was that I updated the Java keystore with certificates
in the part in order to be able to use the https protocol. The
IGNORECERT option can be disabled to ignore certificates.

Looking forward to your »success story«,
Christian


On Mon, Dec 11, 2017 at 9:37 PM, Marco Lettere <m.lett...@gmail.com> wrote:

Hi again,
your words put me on the right way. My doubt was how to pass the information
to the http module send-request functon.
After some investigation I've found out that it may be done by setting some
env variables through -D in the startup scripts pointing to server and
client certificates put in proper keystore and truststore.
At this point http:send-request will transparently use the mutual
certificate exchange without any fiddling with Java code!
That's absolutely great!

Now I'm very close to the final solution and just need to relax the default
hostname verifier. Anyone already did this with basex?

Thanks again for everything.
Marco.

[1] java -cp "$CP" 

Re: [basex-talk] TLS 1.2

2017-12-11 Thread Marco Lettere

Hi again,
your words put me on the right way. My doubt was how to pass the 
information to the http module send-request functon.
After some investigation I've found out that it may be done by setting 
some env variables through -D in the startup scripts pointing to server 
and client certificates put in proper keystore and truststore.
At this point http:send-request will transparently use the mutual 
certificate exchange without any fiddling with Java code!

That's absolutely great!

Now I'm very close to the final solution and just need to relax the 
default hostname verifier. Anyone already did this with basex?


Thanks again for everything.
Marco.

[1] java -cp "$CP" -Djavax.net.ssl.keyStore=/path/to/keystore.p12 
-Djavax.net.ssl.keyStorePassword=thepass 
-Djavax.net.ssl.keyStoreType=PKCS12 
-Djavax.net.ssl.trustStore=/path/to/trustsotre.jks 
-Djavax.net.ssl.trustStorePassword=thepass $BASEX_JVM org.basex.BaseXGUI 
"$@"


On 07/12/2017 22:02, Christian Grün wrote:

Ciao Marco.


Just wanted to be sure that there isn't a way to do everything in XQuery. Is
it?

I think the HTTP Module could be used for that. What have you tried so far?

Saluti
Christian





Re: [basex-talk] TLS 1.2

2017-12-11 Thread Marco Lettere

Hi Christian,
currently not too much since I first have to get used to the protocol 
details of Mutual TLS1.2 authentication [1].
I have a PKCS12 file containing both server and client certificates/keys 
(which are required for mutual auth) and I'm really not sure how all 
this info relataed to the handshake part of the connection rather than 
to the HTTP part itself should be fed into the http-client...

Do you see any possibility around using Java directly for that?
Thank you!
M.

[1] https://en.wikipedia.org/wiki/Mutual_authentication

On 07/12/2017 22:02, Christian Grün wrote:

Ciao Marco.


Just wanted to be sure that there isn't a way to do everything in XQuery. Is
it?

I think the HTTP Module could be used for that. What have you tried so far?

Saluti
Christian





[basex-talk] TLS 1.2

2017-12-07 Thread Marco Lettere

Hi list,

I need to perform an integration with a service that talks HTTP over TLS 
1.2.


I have the mutual certificates and all the rest.

My idea was to do as less as possible (mostly handshaking) in Java and 
then use the Java call the code from inside XQuery where all the rest 
happens.


Just wanted to be sure that there isn't a way to do everything in 
XQuery. Is it?


Thanks,

Marco.



[basex-talk] Documentation typo

2017-11-20 Thread Marco Lettere

Hi all,

I think there is a typo in the documentation on the map:merge function [1].

The Summary states the possibility to use "use-combine" as a value for 
duplicates option but it should be 'combine' only.


Regards,

M.

[1] http://docs.basex.org/wiki/Map_Module#map:merge




Re: [basex-talk] Rounding/parsing decimal vs float

2017-11-01 Thread Marco Lettere
Awesome Leo, thanks!
Similar to what I was thinking about but with a lot or after-thought
optimization!
Great suggestion.
M.

Il 01 nov 2017 1:59 PM, "Leonard Wörteler" <
leonard.woerte...@uni-konstanz.de> ha scritto:

> Hi Marco,
>
> I also do not know of a built-in way to do this, but here is my shot at an
> implementation in XQuery. It uses fast exponentiation [1] and throws a
> custom error if there is more than one "e" in the input.
>
> Hope that helps (or is of interest), I had fun hacking it together,
>   Leo
>
> declare function local:pow($n, $k) {
>   if($k eq 0) then xs:decimal('1')
>   else if($k eq 1) then $n
>   else (
> let $d := $k idiv 2,
> $m := $k mod 2
> return if($m eq 0) then local:pow($n * $n, $d)
> else local:pow($n * $n, $d) * $n
>   )
> };
>
> declare function local:parse-decimal($str as xs:string) as xs:decimal {
>   let $parts := tokenize($str, '[eE]')
>   return switch(count($parts))
> case 1 return xs:decimal($str)
> case 2 return (
>   let $exp  := xs:long($parts[2]),
>   $base := xs:decimal($parts[1]),
>   $b10  := if($exp gt 0) then xs:decimal('10') else
> xs:decimal('0.1')
>   return $base * local:pow($b10, $exp)
> )
> default return error(xs:QName('local:NODECIML'), 'Wrong format: '
> || $str)
> };
>
> local:parse-decimal('123.45e-300')
>
> [1] https://en.wikipedia.org/wiki/Exponentiation_by_squaring
>
> Am Mittwoch, 01. November 2017 13:21 CET, Marco Lettere <
> m.lett...@gmail.com> schrieb:
> > Thanks Kristian,
> > unfortunately this introduces rounding errors which I'm trying to avoid
> > because managing units, measures and monetary amounts.
> > M.
> >
> > Il 01 nov 2017 12:57 PM, "Kristian Kankainen" <krist...@keeleleek.ee> ha
> > scritto:
> >
> > You need to use xs:float or xs:double instead of xs:decimal to be able to
> > use the 'e' or 'E' as the exponent separator.
> >
> > Br,
> > Kristian K1. nov 2017 13:18 kirjutas kuupäeval Marco Lettere <
> > m.lett...@gmail.com>:
> > >
> > > Hi all,
> > >
> > > I thought of asking this in parallel of hacking my own parsing
> procedure
> > > ...
> > >
> > > Is there a native way to parse scientific notation string into
> > > xs:decimal since a direct casting (xs:decimal("1e1")) is not allowed?
> > >
> > > Thanks,
> > >
> > > Marco.
> > >
>
>


Re: [basex-talk] Rounding/parsing decimal vs float

2017-11-01 Thread Marco Lettere
Thanks Kristian,
unfortunately this introduces rounding errors which I'm trying to avoid
because managing units, measures and monetary amounts.
M.

Il 01 nov 2017 12:57 PM, "Kristian Kankainen" <krist...@keeleleek.ee> ha
scritto:

You need to use xs:float or xs:double instead of xs:decimal to be able to
use the 'e' or 'E' as the exponent separator.

Br,
Kristian K1. nov 2017 13:18 kirjutas kuupäeval Marco Lettere <
m.lett...@gmail.com>:
>
> Hi all,
>
> I thought of asking this in parallel of hacking my own parsing procedure
> ...
>
> Is there a native way to parse scientific notation string into
> xs:decimal since a direct casting (xs:decimal("1e1")) is not allowed?
>
> Thanks,
>
> Marco.
>


  1   2   3   >