Re: [basex-talk] differencing the string value of documents

[Graydon]

On Sat, Jun 12, 2021 at 04:23:23PM -0400, Liam R. E. Quin scripsit:
> On Sat, 2021-06-12 at 15:38 -0400, Graydon wrote:
> > This test is meant to test only that no words have been lost or
> > re-ordered; that the transformation is semantically correct is out of
> > scope for it.
> 
> Somerandomwitterings...
> 
> So, i'd probably consider
> (1) make a sequence of words from document A
> 
> Now, if you really hate your CPU :) you could transform A.seq into a
> regular expression,
>   w0.*w1.*w2...
> and match it against the extracted string value of A.

I would have to hate my CPU intensely; some of the real documents run to
a thousand or more pages in PDF.

[snip]
> Doug Lenat i think has written a book around parsing algorithms, as has
> Anne Brüggemann-Klein; Michael Sperberg-McQueen gave a paper at
> Balisage about applications to Schema Validation (or at Extreme
> Markup). Anne's abstraction, whose namei can't remember (sorry), is
> most promising since your problem can be recast as equivalent to
> matching XML Schema grammars to input documents, with the unique
> particle attribution restriction lifted; RelaxNG does this with a hedge
> automaton and that's another approach.

I think this will be helpful in the longer term, since more
general solutions and solutions for whether the transformation is
semantically conformant will be wanted.

(Also likely another few buckets of water will be required. :)

Thanks!

Graydon

-- 
Graydon Saunders  | graydon...@gmail.com
Þæs oferéode, ðisses swá mæg.
-- Deor  ("That passed, so may this.")

Re: [basex-talk] differencing the string value of documents

[Graydon]

On Sat, Jun 12, 2021 at 10:36:13PM +0200, Christian Grün scripsit:
> Thanks. Here’s one way how you could handle that:
[snip]

This is similar to the word-level recursive approach I am currently
taking.  (though there is something to learn in the double function
approach for sure; thank you!)

> Here’s another solution, which avoids recursive calls and the need to
> optimize it for tail calls. It’s based on the handsome built-in
> higher-order function hof:until. Once again, it was Leo (Wörteler) who
> introduced it to BaseX [1]:
> 
> declare function local:correct($old, $new) {
>   let $result := hof:until(
> (: test if there’s something to compare :)
> function($m) {
>   empty($m?old) or empty($m?new)
> },
> (: compare, create new input :)
> function($m) {
>   map {
> 'old': if(head($m?old) = head($m?new))
>   then tail($m?old) else $m?old,
> 'new': tail($m?new)
>   }
> },
> (: initial input :)
> map { 'old': $old, 'new': $new }
>   )
>   (: check if there’s old input left :)
>   return empty($result?old)
> };

This is tremendously cool and I think it will be useful; thank you!

(I also think I am going to have to pour a few buckets of water over my
head in the process of trying to fully comprehend it. :)

Much appreciated!

Graydon

-- 
Graydon Saunders  | graydon...@gmail.com
Þæs oferéode, ðisses swá mæg.
-- Deor  ("That passed, so may this.")

Re: [basex-talk] differencing the string value of documents

[Christian Grün]

Thanks. Here’s one way how you could handle that:

declare function local:correct($old, $new) {
  empty(local:compare($old, $new))
};

declare function local:compare($old, $new) {
  if(head($old) = head($new)) then (
(: skip identical lines :)
local:compare(tail($old), tail($new))
  ) else if(empty($old)) then (
(: old lines are consumed, all fine :)
()
  ) else if(empty($new)) then (
(: old lines remain, bad :)
'sigh'
  ) else (
(: lines differ: skip new line :)
local:compare($old, tail($new))
  )
};

let $old := unparsed-text-lines('before.txt')
for $file in (
  'after-correct.txt',
  'after-fails1.txt',
  'after-fails2.txt'
)
let $new := unparsed-text-lines($file)
return $file || ': ' || local:correct($old, $new)

Here’s another solution, which avoids recursive calls and the need to
optimize it for tail calls. It’s based on the handsome built-in
higher-order function hof:until. Once again, it was Leo (Wörteler) who
introduced it to BaseX [1]:

declare function local:correct($old, $new) {
  let $result := hof:until(
(: test if there’s something to compare :)
function($m) {
  empty($m?old) or empty($m?new)
},
(: compare, create new input :)
function($m) {
  map {
'old': if(head($m?old) = head($m?new))
  then tail($m?old) else $m?old,
'new': tail($m?new)
  }
},
(: initial input :)
map { 'old': $old, 'new': $new }
  )
  (: check if there’s old input left :)
  return empty($result?old)
};

Cheers,
Christian

[1] https://docs.basex.org/wiki/Higher-Order_Functions_Module#hof:until
_

On Sat, Jun 12, 2021 at 9:38 PM Graydon  wrote:
>
> Hello!
>
> On Sat, Jun 12, 2021 at 07:11:00PM +0200, Christian Grün scripsit:
> > Could you share exemplary and minimized input documents with us?
>
> I have created some text documents and attached them.  (The "remember to
> throw away some metadata markup, etc." step on the way to getting text
> to compare from the before and after vocabularies is believed to work
> reliably.)
>
> > Can the structure of the documents (hierarchy of nodes, element names,
> > etc.) be completely ignored?
>
> It can! This test is meant to test only that no words have been lost or
> re-ordered; that the transformation is semantically correct is out of
> scope for it.
>
> Thank you!
> Graydon

Re: [basex-talk] differencing the string value of documents

[Liam R. E. Quin]

On Sat, 2021-06-12 at 15:38 -0400, Graydon wrote:
>  This test is meant to test only that no words have been lost or
> re-ordered; that the transformation is semantically correct is out of
> scope for it.

Somerandomwitterings...

So, i'd probably consider
(1) make a sequence of words from document A

Now, if you really hate your CPU :) you could transform A.seq into a
regular expression,
  w0.*w1.*w2...
and match it against the extracted string value of A.

Starting with ^.*?w0 might reduce the run-time in practice, but the
others all need arbitrary backtracking in case the transformation
introducedone or more  words that occur at that point in the document,
so you have
  w0 w1 w2 w3 w1 w2 w3 w4
to match against
  w0 w1  w2 w3 w3

This could also be written with a recursive function and a helper; the
helper would find the longest match at the current position, and if
that's empty the function returns "nope" and you have to back-track.

Doug Lenat i think has written a book around parsing algorithms, as has
Anne Brüggemann-Klein; Michael Sperberg-McQueen gave a paper at
Balisage about applications to Schema Validation (or at Extreme
Markup). Anne's abstraction, whose namei can't remember (sorry), is
most promising since your problem can be recast as equivalent to
matching XML Schema grammars to input documents, with the unique
particle attribution restriction lifted; RelaxNG does this with a hedge
automaton and that's another approach.


-- 
Liam Quin, https://www.delightfulcomputing.com/
Available for XML/Document/Information Architecture/XSLT/
XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
Barefoot Web-slave, antique illustrations:  http://www.fromoldbooks.org

Re: [basex-talk] differencing the string value of documents

[Graydon]

Hello!

On Sat, Jun 12, 2021 at 07:11:00PM +0200, Christian Grün scripsit:
> Could you share exemplary and minimized input documents with us?

I have created some text documents and attached them.  (The "remember to
throw away some metadata markup, etc." step on the way to getting text
to compare from the before and after vocabularies is believed to work
reliably.)

> Can the structure of the documents (hierarchy of nodes, element names,
> etc.) be completely ignored?

It can! This test is meant to test only that no words have been lost or
re-ordered; that the transformation is semantically correct is out of
scope for it.

Thank you!
Graydon
<>

Re: [basex-talk] differencing the string value of documents

[Christian Grün]

Hi Graydon,

Could you share exemplary and minimized input documents with us?

Can the structure of the documents (hierarchy of nodes, element names,
etc.) be completely ignored?

Best,
Christian


On Sat, Jun 12, 2021 at 6:09 PM Graydon  wrote:
>
> On Fri, Jun 11, 2021 at 09:15:02AM +0200, Martin Honnen scripsit:
> > Of course contains(string(B), string(A)) would alone suffice to check
> > for a partial match.
>
> I think I managed to express the problem badly.
>
> Document A is the original; it has some quantity of text.  Document B is
> produced from Document A via a transformation process that is known to
> add text by doing things like getting display text for an internal link
> from the reference target, inserting boilerplate legal disclaimers,
> inserting standard text ("Chapter 1", etc.), and so on.
>
> The goal is to perform a test to ensure that B still has all of A's
> text, in the same order, after this transformation has taken place.
>
> So no single string compare will work; it will certainly fail, and
> because there are multiple points of insertion in the document (which
> can be of arbitrary length), there's some question of appropriate scale
> of testing.  (Hashing per notional sentences and comparing the hash
> values finds differences but doesn't provide a test for order, for
> example.)
>
> I was hoping that there was a known algorithm for this kind of testing;
> I have trouble believing I'm the first person who has wanted to do it.
>
> It looks like ratcheting through B's text with A's text, one word at a
> time, gives useful results.  It doesn't allow for locating the error
> beyond "something, somewhere, is a problem" but that can be handled
> through diff tools and the full text of each document.
>
> Thanks!
> Graydon
>
>
> --
> Graydon Saunders  | graydon...@gmail.com
> Þæs oferéode, ðisses swá mæg.
> -- Deor  ("That passed, so may this.")

[basex-talk] Permissions

[SW-Service]

Hello,
should perm force authentication and only the user "admin" would be 
authorised?

Best regards Frank

module namespace page = 'http://basex.org/test';
declare
    %rest:path("basex-rest/test/check-rest-perm")
    %perm:allow("admin")
    %rest:GET
    %output:method("xml")
function page:test-check-rest-perm() as element(response) {
    let $response := element response
                    {
                        'Response'
                    }
    return $response
};

Response

Re: [basex-talk] HTTP Client DELETE with body

[Christian Grün]

…and exceptional kudos! ありがとう.


Hans-Juergen Rennau  schrieb am Sa., 12. Juni 2021, 16:56:

> Indeed - common sense & top flexibility!
>
> Am Samstag, 12. Juni 2021, 16:45:05 MESZ hat Marco Lettere <
> m.lett...@gmail.com> Folgendes geschrieben:
>
>
> Thank you! I didn't want to disturb your weekend though Top quality
> BaseX service! 朗
>
> Il sab 12 giu 2021, 16:37 Christian Grün  ha
> scritto:
>
> Grazie, Marco.
>
> You convinced me there is no benefit in keeping the restriction (and
> it we didn’t add it to the draft of version 2 of the spec either [1]).
> I’ve just uploaded a new snapshot [2]; I hope it keeps you going.
>
> Ciao, Christian.
>
> [1] https://expath.github.io/expath-cg/specs/http-client-2/
> [2] https://files.basex.org/releases/latest/
>
>
> > So I guess you’ve stumbled upon an API that expects a payload with
> DELETE?
> >
> > Yes Christian, trying to delete an application role association to a
> user on the Keycloak REST APIs requires the role to be passed as JSON
> object in the DELETE body [1].
> >
> > So at the moment I'm able to add roles to a user but I cannot change
> them because I'm unabale to remove one.
> >
> > Telling just because you asked ;-)
> >
> > Thanks a lot anyway for your attention.
> >
> > [1]
> https://www.keycloak.org/docs-api/11.0/rest-api/index.html#_client_role_mappings_resource
> >
> > Cheers
> > Christian
> >
> > [1] http://expath.org/spec/http-client#d2e104
> >
> >
> >
> >
> > Marco Lettere  schrieb am Sa., 12. Juni 2021,
> 11:48:
> >>
> >> Dear all,
> >>
> >> I have to call an API which model an HTTP DELETE operation based on the
> >> body of the request.
> >>
> >> It looks like currently http:send-request is not allowing body to be
> >> present ("Body not expected for method DELETE.").
> >>
> >> I feel that this is bit too restrictive interpretation of the spec which
> >> states (as far as I recall) that there is no defined semantic for the
> >> operation but not that it is forbidden.
> >>
> >> Is there any possibility to relax this behaviour and allow for bodies in
> >> DELETE calls? [1]
> >>
> >> Thank you very much.
> >>
> >> Marco.
> >>
> >>
> >> [1]
> >>
> https://github.com/BaseXdb/basex/blob/382c8097afb1ec5bac1e9511a2287686c33f669d/basex-core/src/main/java/org/basex/util/http/HttpRequestParser.java
> >>
>
>

Re: [basex-talk] HTTP Client DELETE with body

[Hans-Juergen Rennau]

 Indeed - common sense & top flexibility!

Am Samstag, 12. Juni 2021, 16:45:05 MESZ hat Marco Lettere 
 Folgendes geschrieben:  
 
 Thank you! I didn't want to disturb your weekend though Top quality BaseX 
service! 朗
Il sab 12 giu 2021, 16:37 Christian Grün  ha scritto:

Grazie, Marco.

You convinced me there is no benefit in keeping the restriction (and
it we didn’t add it to the draft of version 2 of the spec either [1]).
I’ve just uploaded a new snapshot [2]; I hope it keeps you going.

Ciao, Christian.

[1] https://expath.github.io/expath-cg/specs/http-client-2/
[2] https://files.basex.org/releases/latest/


> So I guess you’ve stumbled upon an API that expects a payload with DELETE?
>
> Yes Christian, trying to delete an application role association to a user on 
> the Keycloak REST APIs requires the role to be passed as JSON object in the 
> DELETE body [1].
>
> So at the moment I'm able to add roles to a user but I cannot change them 
> because I'm unabale to remove one.
>
> Telling just because you asked ;-)
>
> Thanks a lot anyway for your attention.
>
> [1] 
> https://www.keycloak.org/docs-api/11.0/rest-api/index.html#_client_role_mappings_resource
>
> Cheers
> Christian
>
> [1] http://expath.org/spec/http-client#d2e104
>
>
>
>
> Marco Lettere  schrieb am Sa., 12. Juni 2021, 11:48:
>>
>> Dear all,
>>
>> I have to call an API which model an HTTP DELETE operation based on the
>> body of the request.
>>
>> It looks like currently http:send-request is not allowing body to be
>> present ("Body not expected for method DELETE.").
>>
>> I feel that this is bit too restrictive interpretation of the spec which
>> states (as far as I recall) that there is no defined semantic for the
>> operation but not that it is forbidden.
>>
>> Is there any possibility to relax this behaviour and allow for bodies in
>> DELETE calls? [1]
>>
>> Thank you very much.
>>
>> Marco.
>>
>>
>> [1]
>> https://github.com/BaseXdb/basex/blob/382c8097afb1ec5bac1e9511a2287686c33f669d/basex-core/src/main/java/org/basex/util/http/HttpRequestParser.java
>>

  

Re: [basex-talk] HTTP Client DELETE with body

[Marco Lettere]

Thank you! I didn't want to disturb your weekend though Top quality
BaseX service! 朗

Il sab 12 giu 2021, 16:37 Christian Grün  ha
scritto:

> Grazie, Marco.
>
> You convinced me there is no benefit in keeping the restriction (and
> it we didn’t add it to the draft of version 2 of the spec either [1]).
> I’ve just uploaded a new snapshot [2]; I hope it keeps you going.
>
> Ciao, Christian.
>
> [1] https://expath.github.io/expath-cg/specs/http-client-2/
> [2] https://files.basex.org/releases/latest/
>
>
> > So I guess you’ve stumbled upon an API that expects a payload with
> DELETE?
> >
> > Yes Christian, trying to delete an application role association to a
> user on the Keycloak REST APIs requires the role to be passed as JSON
> object in the DELETE body [1].
> >
> > So at the moment I'm able to add roles to a user but I cannot change
> them because I'm unabale to remove one.
> >
> > Telling just because you asked ;-)
> >
> > Thanks a lot anyway for your attention.
> >
> > [1]
> https://www.keycloak.org/docs-api/11.0/rest-api/index.html#_client_role_mappings_resource
> >
> > Cheers
> > Christian
> >
> > [1] http://expath.org/spec/http-client#d2e104
> >
> >
> >
> >
> > Marco Lettere  schrieb am Sa., 12. Juni 2021,
> 11:48:
> >>
> >> Dear all,
> >>
> >> I have to call an API which model an HTTP DELETE operation based on the
> >> body of the request.
> >>
> >> It looks like currently http:send-request is not allowing body to be
> >> present ("Body not expected for method DELETE.").
> >>
> >> I feel that this is bit too restrictive interpretation of the spec which
> >> states (as far as I recall) that there is no defined semantic for the
> >> operation but not that it is forbidden.
> >>
> >> Is there any possibility to relax this behaviour and allow for bodies in
> >> DELETE calls? [1]
> >>
> >> Thank you very much.
> >>
> >> Marco.
> >>
> >>
> >> [1]
> >>
> https://github.com/BaseXdb/basex/blob/382c8097afb1ec5bac1e9511a2287686c33f669d/basex-core/src/main/java/org/basex/util/http/HttpRequestParser.java
> >>
>

Re: [basex-talk] HTTP Client DELETE with body

[Christian Grün]

Grazie, Marco.

You convinced me there is no benefit in keeping the restriction (and
it we didn’t add it to the draft of version 2 of the spec either [1]).
I’ve just uploaded a new snapshot [2]; I hope it keeps you going.

Ciao, Christian.

[1] https://expath.github.io/expath-cg/specs/http-client-2/
[2] https://files.basex.org/releases/latest/


> So I guess you’ve stumbled upon an API that expects a payload with DELETE?
>
> Yes Christian, trying to delete an application role association to a user on 
> the Keycloak REST APIs requires the role to be passed as JSON object in the 
> DELETE body [1].
>
> So at the moment I'm able to add roles to a user but I cannot change them 
> because I'm unabale to remove one.
>
> Telling just because you asked ;-)
>
> Thanks a lot anyway for your attention.
>
> [1] 
> https://www.keycloak.org/docs-api/11.0/rest-api/index.html#_client_role_mappings_resource
>
> Cheers
> Christian
>
> [1] http://expath.org/spec/http-client#d2e104
>
>
>
>
> Marco Lettere  schrieb am Sa., 12. Juni 2021, 11:48:
>>
>> Dear all,
>>
>> I have to call an API which model an HTTP DELETE operation based on the
>> body of the request.
>>
>> It looks like currently http:send-request is not allowing body to be
>> present ("Body not expected for method DELETE.").
>>
>> I feel that this is bit too restrictive interpretation of the spec which
>> states (as far as I recall) that there is no defined semantic for the
>> operation but not that it is forbidden.
>>
>> Is there any possibility to relax this behaviour and allow for bodies in
>> DELETE calls? [1]
>>
>> Thank you very much.
>>
>> Marco.
>>
>>
>> [1]
>> https://github.com/BaseXdb/basex/blob/382c8097afb1ec5bac1e9511a2287686c33f669d/basex-core/src/main/java/org/basex/util/http/HttpRequestParser.java
>>

Re: [basex-talk] HTTP Client DELETE with body

[Marco Lettere]

So I guess you’ve stumbled upon an API that expects a payload with DELETE?

Yes Christian, trying to delete an application role association to a 
user on the Keycloak REST APIs requires the role to be passed as JSON 
object in the DELETE body [1].


So at the moment I'm able to add roles to a user but I cannot change 
them because I'm unabale to remove one.


Telling just because you asked ;-)

Thanks a lot anyway for your attention.

[1] 
https://www.keycloak.org/docs-api/11.0/rest-api/index.html#_client_role_mappings_resource



Cheers
Christian

[1] http://expath.org/spec/http-client#d2e104 






Marco Lettere mailto:m.lett...@gmail.com>> 
schrieb am Sa., 12. Juni 2021, 11:48:


Dear all,

I have to call an API which model an HTTP DELETE operation based
on the
body of the request.

It looks like currently http:send-request is not allowing body to be
present ("Body not expected for method DELETE.").

I feel that this is bit too restrictive interpretation of the spec
which
states (as far as I recall) that there is no defined semantic for the
operation but not that it is forbidden.

Is there any possibility to relax this behaviour and allow for
bodies in
DELETE calls? [1]

Thank you very much.

Marco.


[1]

https://github.com/BaseXdb/basex/blob/382c8097afb1ec5bac1e9511a2287686c33f669d/basex-core/src/main/java/org/basex/util/http/HttpRequestParser.java

Re: [basex-talk] HTTP Client DELETE with body

[Christian Grün]

Hi Marco,

The error is basically raised because that's required by the spec [1].

But it's right that the RFC2616 doesn't explicitly forbid DELETE requests
with bodies, so we could possibly get rid of the check.

So I guess you’ve stumbled upon an API that expects a payload with DELETE?

Cheers
Christian

[1] http://expath.org/spec/http-client#d2e104




Marco Lettere  schrieb am Sa., 12. Juni 2021, 11:48:

> Dear all,
>
> I have to call an API which model an HTTP DELETE operation based on the
> body of the request.
>
> It looks like currently http:send-request is not allowing body to be
> present ("Body not expected for method DELETE.").
>
> I feel that this is bit too restrictive interpretation of the spec which
> states (as far as I recall) that there is no defined semantic for the
> operation but not that it is forbidden.
>
> Is there any possibility to relax this behaviour and allow for bodies in
> DELETE calls? [1]
>
> Thank you very much.
>
> Marco.
>
>
> [1]
>
> https://github.com/BaseXdb/basex/blob/382c8097afb1ec5bac1e9511a2287686c33f669d/basex-core/src/main/java/org/basex/util/http/HttpRequestParser.java
>
>

[basex-talk] HTTP Client DELETE with body

[Marco Lettere]

Dear all,

I have to call an API which model an HTTP DELETE operation based on the 
body of the request.


It looks like currently http:send-request is not allowing body to be 
present ("Body not expected for method DELETE.").


I feel that this is bit too restrictive interpretation of the spec which 
states (as far as I recall) that there is no defined semantic for the 
operation but not that it is forbidden.


Is there any possibility to relax this behaviour and allow for bodies in 
DELETE calls? [1]


Thank you very much.

Marco.


[1] 
https://github.com/BaseXdb/basex/blob/382c8097afb1ec5bac1e9511a2287686c33f669d/basex-core/src/main/java/org/basex/util/http/HttpRequestParser.java