[basex-talk] How to improve performance of join query

[Chuck Bearden]

Hi all,

I'm joining elements from two sets of documents in a single database.
There are many elements (six figures) in each set. The query doesn't
finish after several minutes. My suspicion is that the indices aren't
being used, except to limit the fingerprint 'content' elements to
those with "Person" as the value of 'contentFamily'. Perhaps my data
and query demand some form of optimization that hasn't been
implemented yet, but perhaps there is something I could do to fix the
query. I'm using BaseX 8.3.0.

The query:
-
xquery version "3.0";
declare namespace
fingerprint="http://atira.dk/schemas/pure4/wsdl/template/fingerprint/current";;
declare namespace
concept="http://atira.dk/schemas/pure4/wsdl/template/concept/current";;
declare namespace core="http://atira.dk/schemas/pure4/model/core/current";;
declare namespace xsi="http://www.w3.org/2001/XMLSchema-instance";;
declare namespace
fincur="http://atira.dk/schemas/pure4/model/template/fingerprint/current";;
declare namespace
concur="http://atira.dk/schemas/pure4/model/template/concept/current";;

declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization";;
declare option output:method "csv";
declare option output:csv "header=yes, separator=comma";

let $coll := collection('pure_20151109')
let $concept_contents :=
$coll/concept:GetConceptResponse/core:result/core:content
let $fingerprint_contents :=
$coll/fingerprint:GetFingerprintResponse/core:result/core:content

for $fin_con in $fingerprint_contents[fincur:contentFamily/text()="Person"]
  /fincur:rankedConcepts/fincur:rankedConcept
let $concept_uuid := $fin_con/fincur:conceptUuid
let $rank := $fin_con/fincur:rank
let $weighted_rank := $fin_con/fincur:weightedRank
let $person_uuid := $fin_con/ancestor::core:content[1]/@uuid
let $concept_name :=
$concept_contents[@uuid=$concept_uuid]/concur:name/core:localizedString
return

  
{data($person_uuid)}
{$concept_uuid/text()}
{$rank/text()}
{$weighted_rank/text()}
{$concept_name/text()}
  

-

The query info from the query, which I stopped after 3 minutes:
-
Error:
Interrupted.
Compiling:
- pre-evaluating collection("pure_20151109")
- inlining $coll_0
- inlining $fingerprint_contents_2
- applying text index for "Person"
- inlining $rank_5
- inlining $weighted_rank_6
- inlining $person_uuid_7
- inlining $concept_name_8
Query:
xquery version "3.0"; declare namespace
fingerprint="http://atira.dk/schemas/pure4/wsdl/template/fingerprint/current";;
declare namespace
concept="http://atira.dk/schemas/pure4/wsdl/template/concept/current";;
declare namespace
core="http://atira.dk/schemas/pure4/model/core/current";; declare
namespace xsi="http://www.w3.org/2001/XMLSchema-instance";; declare
namespace 
fincur="http://atira.dk/schemas/pure4/model/template/fingerprint/current";;
declare namespace
concur="http://atira.dk/schemas/pure4/model/template/concept/current";;
declare namespace output =
"http://www.w3.org/2010/xslt-xquery-serialization";; declare option
output:method "csv"; declare option output:csv "header=yes,
separator=comma"; let $coll := collection('pure_20151109') let
$concept_contents :=
$coll/concept:GetConceptResponse/core:result/core:content let
$fingerprint_contents :=
$coll/fingerprint:GetFingerprintResponse/core:result/core:content for
$fin_con in $fingerprint_contents[fincur:contentFamily/text()="Person"]
/fincur:rankedConcepts/fincur:rankedConcept let $concept_uuid :=
$fin_con/fincur:conceptUuid let $rank := $fin_con/fincur:rank let
$weighted_rank := $fin_con/fincur:weightedRank let $person_uuid :=
$fin_con/ancestor::core:content[1]/@uuid let $concept_name :=
$concept_contents[@uuid=$concept_uuid]/concur:name/core:localizedString
return   {data($person_uuid)}
{$concept_uuid/text()}
{$rank/text()}
{$weighted_rank/text()}
{$concept_name/text()}  
Optimized Query:
let $concept_contents_1 := (db:open-pre("pure_20151109",0),
...)/concept:GetConceptResponse/core:result/core:content for
$fin_con_3 in db:text("pure_20151109",
"Person")/parent::fincur:contentFamily/parent::core:content[parent::core:result/parent::fingerprint:GetFingerprintResponse/parent::document-node()]/fincur:rankedConcepts/fincur:rankedConcept
let $concept_uuid_4 := $fin_con_3/fincur:conceptUuid return element
csv { (element record { (element person_uuid {
(data($fin_con_3/ancestor::core:content[position() = 1]/@uuid)) },
element concept_uuid { ($concept_uuid_4/text()) }, element rank {
($fin_con_3/fincur:rank/text()) }, element weighted_rank {
($fin_con_3/fincur:weightedRank/text()) }, element concept_name {
(($concept_contents_1)[(@uuid =
$concept_uuid_4)]/concur:name/core:localizedString/text()) }) }) }
Query plan:

  

  
  

  
  
  
  
  




  


  
  

  



  



  
 

Re: [basex-talk] sending variables to xquery module via client api (python)

[Cholfin, Bryan]

Yes, that worked—but I think I solved the problem (I wouldn’t have seen it
except for trying this, though). And of course it is trivial:

I had originally:

session.execute(“SET BINDINGS journal_id=‘hc2’”)

because I was following the example in the wiki. Changing that to:

session.execute(“SET BINDINGS journal_id=hc2”)


worked. Of course, assigning a variable and composing the command string
by concatenation also works, so I shot myself in the foot by thinking I
could just use a fixed string to test.


So thanks for the advice—it helped.


Bryan Cholfin
Associate Editor for Digital Projects and Metadata
Harvard University Press
617.496.9116
79 Garden Street | Cambridge, MA 02138-1400
www.hup.harvard.edu 




On 11/17/15, 2:29 PM, "Christian Grün"  wrote:

>> No—trace doesn’t seem to send anything to stderr. (I even forced a
>>couple
>> of errors just to make sure stderr was going where I thought it was).
>
>This would indicate that the predicate is never evaluated.
>
>How does your full query look like? Do you get output when running the
>query on command-line?..
>
>  basex -b ext_variable=xyz query.xq
>
>
>
>
>
>>
>>
>> Bryan Cholfin
>> Associate Editor for Digital Projects and Metadata
>> Harvard University Press
>> 617.496.9116
>> 79 Garden Street | Cambridge, MA 02138-1400
>> www.hup.harvard.edu 
>>
>>
>>
>>
>> On 11/17/15, 1:59 PM, "Christian Grün" 
>>wrote:
>>
 I get this:
 [...]
>>>
>>>This looks like the output that’s stored in the database logs, while
>>>trace output is written to STDERR (not STDOUT, as I wrote in my last
>>>response, sorry). Do you see anything there?
>>>
>>>
>>>
>>>
 13:50:53.282127.0.0.1:39813 admin   REQUEST SET BINDINGS
 journal_id='hc2'1.02 ms
 13:50:53.282127.0.0.1:39813 admin   OK  BINDINGS:
journal_id='hc2'  0.11 ms
 13:50:53.284127.0.0.1:39813 admin   REQUEST RUN
 /home/bryan/basex/webapp/waldo/waldotext.xqm1.86 ms
 13:50:53.287127.0.0.1:39813 admin   OK  Query "waldotext.xqm"
executed in
 3.2 ms. 3.63 ms
 13:50:53.288127.0.0.1:39813 admin   REQUEST EXIT0.16 ms
 13:50:53.288127.0.0.1:39813 admin   OK  0.09 ms


 When I run the module from the gui and set the variable manually, it
works
 as I expect.


 Bryan Cholfin
 Associate Editor for Digital Projects and Metadata
 Harvard University Press
 617.496.9116
 79 Garden Street | Cambridge, MA 02138-1400
 www.hup.harvard.edu 




 On 11/17/15, 1:19 PM, "Christian Grün" 
wrote:

>> declare variable $ext_variable as xs:string external;
>> for $div in collection(Œwaldo¹)//div[@xml:id=$ext_variable]
>
>What is output on server STDOUT if you trace the variable?..
>
>  declare variable $ext_variable as xs:string external;
>  for $div in collection(Œwaldo¹)//div[@xml:id=trace($ext_variable)]
>
>
>>
>> I get nothing. And yet I¹ve been able to return the variable itself
>>from the
>> module and check its type, and use it in a let assignment. So I can
>>see
>>that
>> the module is receiving the value I send, it just isn¹t matching for
>>some
>> reason.
>>
>> Bryan Cholfin
>>
>> Associate Editor for Digital Projects and Metadata
>> Harvard University Press
>> 617.496.9116
>> 79 Garden Street | Cambridge, MA 02138-1400
>> www.hup.harvard.edu
>>
>>
>> From: Etanchaud Fabrice 
>> Date: Tuesday, November 17, 2015 at 4:08 AM
>> To: "Cholfin, Bryan" ,
>> "basex-talk@mailman.uni-konstanz.de"
>>
>> Subject: RE: [basex-talk] sending variables to xquery module via
>>client
>>api
>> (python)
>>
>> Hi Bryan,
>>
>>
>>
>> Could http://docs.basex.org/wiki/Options#BINDINGSgive you a
>>solution ?
>>
>>
>>
>> Best regards,
>>
>> Fabrice
>>
>>
>>
>> De : basex-talk-boun...@mailman.uni-konstanz.de
>> [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de
>>Cholfin,
>> Bryan
>> Envoyé : lundi 16 novembre 2015 23:08
>> À : basex-talk@mailman.uni-konstanz.de
>> Objet : [basex-talk] sending variables to xquery module via client
>>api
>> (python)
>>
>>
>>
>> Hi‹
>>
>>
>>
>> Hoping someone can help me with this. I¹m trying to write a python
>>script
>> that calls an xquery module in the database. It seems to be pretty
>>easy
>>to
>> use a line such as:
>>
>>
>>
>> session.execute(³run gettext.xqm²)
>>
>>
>>
>> and get back the response I want. What I want to do is pass a bound
>>variable
>> to the module. The client api has a provision to bind a variable to
>>a
>>query
>> object, but I¹m not making a query object. I¹m simply 

Re: [basex-talk] sending variables to xquery module via client api (python)

[Christian Grün]

> No—trace doesn’t seem to send anything to stderr. (I even forced a couple
> of errors just to make sure stderr was going where I thought it was).

This would indicate that the predicate is never evaluated.

How does your full query look like? Do you get output when running the
query on command-line?..

  basex -b ext_variable=xyz query.xq





>
>
> Bryan Cholfin
> Associate Editor for Digital Projects and Metadata
> Harvard University Press
> 617.496.9116
> 79 Garden Street | Cambridge, MA 02138-1400
> www.hup.harvard.edu 
>
>
>
>
> On 11/17/15, 1:59 PM, "Christian Grün"  wrote:
>
>>> I get this:
>>> [...]
>>
>>This looks like the output that’s stored in the database logs, while
>>trace output is written to STDERR (not STDOUT, as I wrote in my last
>>response, sorry). Do you see anything there?
>>
>>
>>
>>
>>> 13:50:53.282127.0.0.1:39813 admin   REQUEST SET BINDINGS
>>> journal_id='hc2'1.02 ms
>>> 13:50:53.282127.0.0.1:39813 admin   OK  BINDINGS:
>>>journal_id='hc2'  0.11 ms
>>> 13:50:53.284127.0.0.1:39813 admin   REQUEST RUN
>>> /home/bryan/basex/webapp/waldo/waldotext.xqm1.86 ms
>>> 13:50:53.287127.0.0.1:39813 admin   OK  Query "waldotext.xqm"
>>>executed in
>>> 3.2 ms. 3.63 ms
>>> 13:50:53.288127.0.0.1:39813 admin   REQUEST EXIT0.16 ms
>>> 13:50:53.288127.0.0.1:39813 admin   OK  0.09 ms
>>>
>>>
>>> When I run the module from the gui and set the variable manually, it
>>>works
>>> as I expect.
>>>
>>>
>>> Bryan Cholfin
>>> Associate Editor for Digital Projects and Metadata
>>> Harvard University Press
>>> 617.496.9116
>>> 79 Garden Street | Cambridge, MA 02138-1400
>>> www.hup.harvard.edu 
>>>
>>>
>>>
>>>
>>> On 11/17/15, 1:19 PM, "Christian Grün" 
>>>wrote:
>>>
> declare variable $ext_variable as xs:string external;
> for $div in collection(Œwaldo¹)//div[@xml:id=$ext_variable]

What is output on server STDOUT if you trace the variable?..

  declare variable $ext_variable as xs:string external;
  for $div in collection(Œwaldo¹)//div[@xml:id=trace($ext_variable)]


>
> I get nothing. And yet I¹ve been able to return the variable itself
>from the
> module and check its type, and use it in a let assignment. So I can
>see
>that
> the module is receiving the value I send, it just isn¹t matching for
>some
> reason.
>
> Bryan Cholfin
>
> Associate Editor for Digital Projects and Metadata
> Harvard University Press
> 617.496.9116
> 79 Garden Street | Cambridge, MA 02138-1400
> www.hup.harvard.edu
>
>
> From: Etanchaud Fabrice 
> Date: Tuesday, November 17, 2015 at 4:08 AM
> To: "Cholfin, Bryan" ,
> "basex-talk@mailman.uni-konstanz.de"
>
> Subject: RE: [basex-talk] sending variables to xquery module via
>client
>api
> (python)
>
> Hi Bryan,
>
>
>
> Could http://docs.basex.org/wiki/Options#BINDINGSgive you a solution ?
>
>
>
> Best regards,
>
> Fabrice
>
>
>
> De : basex-talk-boun...@mailman.uni-konstanz.de
> [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de
>Cholfin,
> Bryan
> Envoyé : lundi 16 novembre 2015 23:08
> À : basex-talk@mailman.uni-konstanz.de
> Objet : [basex-talk] sending variables to xquery module via client api
> (python)
>
>
>
> Hi‹
>
>
>
> Hoping someone can help me with this. I¹m trying to write a python
>script
> that calls an xquery module in the database. It seems to be pretty
>easy
>to
> use a line such as:
>
>
>
> session.execute(³run gettext.xqm²)
>
>
>
> and get back the response I want. What I want to do is pass a bound
>variable
> to the module. The client api has a provision to bind a variable to a
>query
> object, but I¹m not making a query object. I¹m simply executing the
>stored
> query.  I don¹t find anything in the documentation to suggest that a
> variable can be added to the run command this way.
>
>
>
> Bryan Cholfin
>
> Associate Editor for Digital Projects and Metadata
> Harvard University Press
> 617.496.9116
> 79 Garden Street | Cambridge, MA 02138-1400
> www.hup.harvard.edu
>>>
>

Re: [basex-talk] sending variables to xquery module via client api (python)

[Cholfin, Bryan]

No—trace doesn’t seem to send anything to stderr. (I even forced a couple
of errors just to make sure stderr was going where I thought it was).


Bryan Cholfin
Associate Editor for Digital Projects and Metadata
Harvard University Press
617.496.9116
79 Garden Street | Cambridge, MA 02138-1400
www.hup.harvard.edu 




On 11/17/15, 1:59 PM, "Christian Grün"  wrote:

>> I get this:
>> [...]
>
>This looks like the output that’s stored in the database logs, while
>trace output is written to STDERR (not STDOUT, as I wrote in my last
>response, sorry). Do you see anything there?
>
>
>
>
>> 13:50:53.282127.0.0.1:39813 admin   REQUEST SET BINDINGS
>> journal_id='hc2'1.02 ms
>> 13:50:53.282127.0.0.1:39813 admin   OK  BINDINGS:
>>journal_id='hc2'  0.11 ms
>> 13:50:53.284127.0.0.1:39813 admin   REQUEST RUN
>> /home/bryan/basex/webapp/waldo/waldotext.xqm1.86 ms
>> 13:50:53.287127.0.0.1:39813 admin   OK  Query "waldotext.xqm"
>>executed in
>> 3.2 ms. 3.63 ms
>> 13:50:53.288127.0.0.1:39813 admin   REQUEST EXIT0.16 ms
>> 13:50:53.288127.0.0.1:39813 admin   OK  0.09 ms
>>
>>
>> When I run the module from the gui and set the variable manually, it
>>works
>> as I expect.
>>
>>
>> Bryan Cholfin
>> Associate Editor for Digital Projects and Metadata
>> Harvard University Press
>> 617.496.9116
>> 79 Garden Street | Cambridge, MA 02138-1400
>> www.hup.harvard.edu 
>>
>>
>>
>>
>> On 11/17/15, 1:19 PM, "Christian Grün" 
>>wrote:
>>
 declare variable $ext_variable as xs:string external;
 for $div in collection(Œwaldo¹)//div[@xml:id=$ext_variable]
>>>
>>>What is output on server STDOUT if you trace the variable?..
>>>
>>>  declare variable $ext_variable as xs:string external;
>>>  for $div in collection(Œwaldo¹)//div[@xml:id=trace($ext_variable)]
>>>
>>>

 I get nothing. And yet I¹ve been able to return the variable itself
from the
 module and check its type, and use it in a let assignment. So I can
see
that
 the module is receiving the value I send, it just isn¹t matching for
some
 reason.

 Bryan Cholfin

 Associate Editor for Digital Projects and Metadata
 Harvard University Press
 617.496.9116
 79 Garden Street | Cambridge, MA 02138-1400
 www.hup.harvard.edu


 From: Etanchaud Fabrice 
 Date: Tuesday, November 17, 2015 at 4:08 AM
 To: "Cholfin, Bryan" ,
 "basex-talk@mailman.uni-konstanz.de"

 Subject: RE: [basex-talk] sending variables to xquery module via
client
api
 (python)

 Hi Bryan,



 Could http://docs.basex.org/wiki/Options#BINDINGSgive you a solution ?



 Best regards,

 Fabrice



 De : basex-talk-boun...@mailman.uni-konstanz.de
 [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de
Cholfin,
 Bryan
 Envoyé : lundi 16 novembre 2015 23:08
 À : basex-talk@mailman.uni-konstanz.de
 Objet : [basex-talk] sending variables to xquery module via client api
 (python)



 Hi‹



 Hoping someone can help me with this. I¹m trying to write a python
script
 that calls an xquery module in the database. It seems to be pretty
easy
to
 use a line such as:



 session.execute(³run gettext.xqm²)



 and get back the response I want. What I want to do is pass a bound
variable
 to the module. The client api has a provision to bind a variable to a
query
 object, but I¹m not making a query object. I¹m simply executing the
stored
 query.  I don¹t find anything in the documentation to suggest that a
 variable can be added to the run command this way.



 Bryan Cholfin

 Associate Editor for Digital Projects and Metadata
 Harvard University Press
 617.496.9116
 79 Garden Street | Cambridge, MA 02138-1400
 www.hup.harvard.edu
>>

Re: [basex-talk] sending variables to xquery module via client api (python)

[Christian Grün]

> I get this:
> [...]

This looks like the output that’s stored in the database logs, while
trace output is written to STDERR (not STDOUT, as I wrote in my last
response, sorry). Do you see anything there?




> 13:50:53.282127.0.0.1:39813 admin   REQUEST SET BINDINGS
> journal_id='hc2'1.02 ms
> 13:50:53.282127.0.0.1:39813 admin   OK  BINDINGS: journal_id='hc2'
>   0.11 ms
> 13:50:53.284127.0.0.1:39813 admin   REQUEST RUN
> /home/bryan/basex/webapp/waldo/waldotext.xqm1.86 ms
> 13:50:53.287127.0.0.1:39813 admin   OK  Query "waldotext.xqm" 
> executed in
> 3.2 ms. 3.63 ms
> 13:50:53.288127.0.0.1:39813 admin   REQUEST EXIT0.16 ms
> 13:50:53.288127.0.0.1:39813 admin   OK  0.09 ms
>
>
> When I run the module from the gui and set the variable manually, it works
> as I expect.
>
>
> Bryan Cholfin
> Associate Editor for Digital Projects and Metadata
> Harvard University Press
> 617.496.9116
> 79 Garden Street | Cambridge, MA 02138-1400
> www.hup.harvard.edu 
>
>
>
>
> On 11/17/15, 1:19 PM, "Christian Grün"  wrote:
>
>>> declare variable $ext_variable as xs:string external;
>>> for $div in collection(Œwaldo¹)//div[@xml:id=$ext_variable]
>>
>>What is output on server STDOUT if you trace the variable?..
>>
>>  declare variable $ext_variable as xs:string external;
>>  for $div in collection(Œwaldo¹)//div[@xml:id=trace($ext_variable)]
>>
>>
>>>
>>> I get nothing. And yet I¹ve been able to return the variable itself
>>>from the
>>> module and check its type, and use it in a let assignment. So I can see
>>>that
>>> the module is receiving the value I send, it just isn¹t matching for
>>>some
>>> reason.
>>>
>>> Bryan Cholfin
>>>
>>> Associate Editor for Digital Projects and Metadata
>>> Harvard University Press
>>> 617.496.9116
>>> 79 Garden Street | Cambridge, MA 02138-1400
>>> www.hup.harvard.edu
>>>
>>>
>>> From: Etanchaud Fabrice 
>>> Date: Tuesday, November 17, 2015 at 4:08 AM
>>> To: "Cholfin, Bryan" ,
>>> "basex-talk@mailman.uni-konstanz.de"
>>>
>>> Subject: RE: [basex-talk] sending variables to xquery module via client
>>>api
>>> (python)
>>>
>>> Hi Bryan,
>>>
>>>
>>>
>>> Could http://docs.basex.org/wiki/Options#BINDINGSgive you a solution ?
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Fabrice
>>>
>>>
>>>
>>> De : basex-talk-boun...@mailman.uni-konstanz.de
>>> [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de
>>>Cholfin,
>>> Bryan
>>> Envoyé : lundi 16 novembre 2015 23:08
>>> À : basex-talk@mailman.uni-konstanz.de
>>> Objet : [basex-talk] sending variables to xquery module via client api
>>> (python)
>>>
>>>
>>>
>>> Hi‹
>>>
>>>
>>>
>>> Hoping someone can help me with this. I¹m trying to write a python
>>>script
>>> that calls an xquery module in the database. It seems to be pretty easy
>>>to
>>> use a line such as:
>>>
>>>
>>>
>>> session.execute(³run gettext.xqm²)
>>>
>>>
>>>
>>> and get back the response I want. What I want to do is pass a bound
>>>variable
>>> to the module. The client api has a provision to bind a variable to a
>>>query
>>> object, but I¹m not making a query object. I¹m simply executing the
>>>stored
>>> query.  I don¹t find anything in the documentation to suggest that a
>>> variable can be added to the run command this way.
>>>
>>>
>>>
>>> Bryan Cholfin
>>>
>>> Associate Editor for Digital Projects and Metadata
>>> Harvard University Press
>>> 617.496.9116
>>> 79 Garden Street | Cambridge, MA 02138-1400
>>> www.hup.harvard.edu
>

Re: [basex-talk] sending variables to xquery module via client api (python)

[Cholfin, Bryan]

I get this:

13:50:53.282127.0.0.1:39813 admin   REQUEST SET BINDINGS
journal_id='hc2'1.02 ms
13:50:53.282127.0.0.1:39813 admin   OK  BINDINGS: journal_id='hc2'  
0.11 ms
13:50:53.284127.0.0.1:39813 admin   REQUEST RUN
/home/bryan/basex/webapp/waldo/waldotext.xqm1.86 ms
13:50:53.287127.0.0.1:39813 admin   OK  Query "waldotext.xqm" executed 
in
3.2 ms. 3.63 ms
13:50:53.288127.0.0.1:39813 admin   REQUEST EXIT0.16 ms
13:50:53.288127.0.0.1:39813 admin   OK  0.09 ms


When I run the module from the gui and set the variable manually, it works
as I expect. 


Bryan Cholfin
Associate Editor for Digital Projects and Metadata
Harvard University Press
617.496.9116
79 Garden Street | Cambridge, MA 02138-1400
www.hup.harvard.edu 




On 11/17/15, 1:19 PM, "Christian Grün"  wrote:

>> declare variable $ext_variable as xs:string external;
>> for $div in collection(Œwaldo¹)//div[@xml:id=$ext_variable]
>
>What is output on server STDOUT if you trace the variable?..
>
>  declare variable $ext_variable as xs:string external;
>  for $div in collection(Œwaldo¹)//div[@xml:id=trace($ext_variable)]
>
>
>>
>> I get nothing. And yet I¹ve been able to return the variable itself
>>from the
>> module and check its type, and use it in a let assignment. So I can see
>>that
>> the module is receiving the value I send, it just isn¹t matching for
>>some
>> reason.
>>
>> Bryan Cholfin
>>
>> Associate Editor for Digital Projects and Metadata
>> Harvard University Press
>> 617.496.9116
>> 79 Garden Street | Cambridge, MA 02138-1400
>> www.hup.harvard.edu
>>
>>
>> From: Etanchaud Fabrice 
>> Date: Tuesday, November 17, 2015 at 4:08 AM
>> To: "Cholfin, Bryan" ,
>> "basex-talk@mailman.uni-konstanz.de"
>>
>> Subject: RE: [basex-talk] sending variables to xquery module via client
>>api
>> (python)
>>
>> Hi Bryan,
>>
>>
>>
>> Could http://docs.basex.org/wiki/Options#BINDINGSgive you a solution ?
>>
>>
>>
>> Best regards,
>>
>> Fabrice
>>
>>
>>
>> De : basex-talk-boun...@mailman.uni-konstanz.de
>> [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de
>>Cholfin,
>> Bryan
>> Envoyé : lundi 16 novembre 2015 23:08
>> À : basex-talk@mailman.uni-konstanz.de
>> Objet : [basex-talk] sending variables to xquery module via client api
>> (python)
>>
>>
>>
>> Hi‹
>>
>>
>>
>> Hoping someone can help me with this. I¹m trying to write a python
>>script
>> that calls an xquery module in the database. It seems to be pretty easy
>>to
>> use a line such as:
>>
>>
>>
>> session.execute(³run gettext.xqm²)
>>
>>
>>
>> and get back the response I want. What I want to do is pass a bound
>>variable
>> to the module. The client api has a provision to bind a variable to a
>>query
>> object, but I¹m not making a query object. I¹m simply executing the
>>stored
>> query.  I don¹t find anything in the documentation to suggest that a
>> variable can be added to the run command this way.
>>
>>
>>
>> Bryan Cholfin
>>
>> Associate Editor for Digital Projects and Metadata
>> Harvard University Press
>> 617.496.9116
>> 79 Garden Street | Cambridge, MA 02138-1400
>> www.hup.harvard.edu

Re: [basex-talk] Fwd: Bash command

[Christian Grün]

Hi Joseph,

It shouldn’t have anything to do with RESTXQ (RESTXQ is nothing else
than a plain API for server middleware). I assume that your local and
the server machine use different default encodings, which is why you
need to specify different encodings with proc:execute.

Hope this helps,
Christian


> I have been able to solve my problem within my local machine by specifying 
> ISO-8859-1 in proc:execute. In this case the bash command called in a RESTXQ 
> function is able to return the right characters. However it not clear to me 
> why I need to specify that encoding (instead of utf8) in order to get the 
> Greek characters dislpayed.
>
> Unfortunately, however, this does not work on my server. I checked that the 
> bash has utf8 and indeed if I run the same scripts there results are properly 
> displayed. However, within RESTXQ the same command (called via proc:execute) 
> only returns question marks as a result.
>
> Do you think this can be due to a problem inside RESTXQ or something else?
>
> Thanks,
> Joseph
>
> Il giorno 09/nov/2015, alle ore 15:18, Jens Erat  
> ha scritto:
>
> Please report the encoding of your actual files. What is the output of
> `env | grep LANG` (when stored in your bash script)? If you pipe the
> output in another file, what will `file -i [output-file]` report (`file`
> applies some magic to discover file formats and encoding)?
>
>> Am 09.11.2015 um 14:06 schrieb meumapple:
>> Hi All,
>>
>> I have been able to run the command just putting it in a bash file and then 
>> running this file with proc:execute. It works. The problem now is with the 
>> encoding of the text returned by proc:execute. In my bash I can display 
>> Greek characters, but they are messed up if returned by the proc:execute 
>> function. I also added utf-8 as a parameter for the function but nothing 
>> changes. How can I solve this? Thanks.
>>
>> Joseph
>>
>> Il giorno 09/nov/2015, alle ore 10:41, Jens Erat  
>> ha scritto:
>>
>> Hi Joseph,
>>
>> As far as I understand the documentation, BaseX `proc:system` does not
>> use a shell interpreter by default. `<` or also the pipe `|` are
>> features offered by your shell, not the operating system.
>>
>> Instead of passing a single parameter, pass the command as parameter to
>> `/usr/bin/env sh -c`, which will make BaseX run a shell that interpretes
>> I/O indirection:
>>
>>   proc:system('/usr/bin/env',  ('sh', '-c', 'opennlp postagger
>> model.bin < file.txt'))
>>
>> This might be a reasonable thing to provide as a built-in function, I'm
>> also not sure whether one can connect some kind of stream/strings to
>> STDIN, STDOUT and STDERR.
>>
>> Regards from Lake Constance, Germany,
>> Jens
>>
>> Am 09.11.2015 um 10:19 schrieb Christian Grün:
 Any suggestions?
>>>
>>> Any example? ;)
>>> Christian
>>>
>>>
>>>
>>> Thanks.

 Joseph


 Inizio messaggio inoltrato:

 Da: meumapple 
 Data: 06 novembre 2015 20:52:08 CET
 A: BaseX 
 Oggetto: Bash command

 Hi all,

 I am using proc:system for a bash command but I cannot run it. It looks 
 like
 opennlp postagger model.bin < file.txt. I suspect that the problem is the
 less-than sign, which is converted into < Is there any way to keep the
 less-than sign and pass it to the bash properly? Thanks.

 Joseph
>
> --
> Jens Erat
> Universität Konstanz
> Kommunikations-, Infomations-, Medienzentrum (KIM)
> Abteilung Basisdienste
> D-78457 Konstanz
> Mail: jens.e...@uni-konstanz.de
>

Re: [basex-talk] zip:update-entries - unexpected results

[Christian Grün]

We didn’t spend energy in the ZIP Module for a long time. Did you try
to use the Archive Module?


On Mon, Nov 16, 2015 at 5:06 PM, Hondros, Constantine (ELS-AMS)
 wrote:
> Hi all,
>
>
>
> I’m attempting to create a modified copy a zipfile using zip:update-entries,
> but my output is a zip structure with zero-byte files. Am I missing a step?
> I’ve followed the example in:
> http://docs.basex.org/wiki/ZIP_Module#zip:update-entries
>
>
>
> Code snippet as follows:
>
>
>
> declare function util:clean-zip($zip as xs:string, $out as xs:string) {
>
>   copy $doc := zip:entries($zip)
>
>   modify delete node $doc//zip:entry[ends-with(lower-case(@name), '.pdf')]
>
>   return zip:update-entries($doc, $out)
>
> };
>
>
>
> Thanks for any pointers ;-)
>
>
>
> C.
>
>
> 
>
> Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
> Netherlands, Registration No. 33156677, Registered in The Netherlands.

Re: [basex-talk] sending variables to xquery module via client api (python)

[Cholfin, Bryan]

That almost works. It appears I can pass the variable by executing the 'set 
bindings' command ahead of the query. But if I use the variable in a test like 
this:

declare variable $ext_variable as xs:string external;
for $div in collection('waldo')//div[@xml:id=$ext_variable]

I get nothing. And yet I've been able to return the variable itself from the 
module and check its type, and use it in a let assignment. So I can see that 
the module is receiving the value I send, it just isn't matching for some 
reason.

Bryan Cholfin

Associate Editor for Digital Projects and Metadata
Harvard University Press
617.496.9116
79 Garden Street | Cambridge, MA 02138-1400
www.hup.harvard.edu

From: Etanchaud Fabrice 
mailto:fabrice.etanch...@horanet.com>>
Date: Tuesday, November 17, 2015 at 4:08 AM
To: "Cholfin, Bryan" 
mailto:bryan_chol...@harvard.edu>>, 
"basex-talk@mailman.uni-konstanz.de" 
mailto:basex-talk@mailman.uni-konstanz.de>>
Subject: RE: [basex-talk] sending variables to xquery module via client api 
(python)

Hi Bryan,

Could 
http://docs.basex.org/wiki/Options#BINDINGSgive
 you a solution ?

Best regards,
Fabrice

De : 
basex-talk-boun...@mailman.uni-konstanz.de
 [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Cholfin, 
Bryan
Envoyé : lundi 16 novembre 2015 23:08
À : 
basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] sending variables to xquery module via client api (python)

Hi-

Hoping someone can help me with this. I'm trying to write a python script that 
calls an xquery module in the database. It seems to be pretty easy to use a 
line such as:

session.execute("run gettext.xqm")

and get back the response I want. What I want to do is pass a bound variable to 
the module. The client api has a provision to bind a variable to a query 
object, but I'm not making a query object. I'm simply executing the stored 
query.  I don't find anything in the documentation to suggest that a variable 
can be added to the run command this way.

Bryan Cholfin

Associate Editor for Digital Projects and Metadata
Harvard University Press
617.496.9116
79 Garden Street | Cambridge, MA 02138-1400
www.hup.harvard.edu

Re: [basex-talk] sending variables to xquery module via client api (python)

[Christian Grün]

> declare variable $ext_variable as xs:string external;
> for $div in collection(‘waldo’)//div[@xml:id=$ext_variable]

What is output on server STDOUT if you trace the variable?..

  declare variable $ext_variable as xs:string external;
  for $div in collection(‘waldo’)//div[@xml:id=trace($ext_variable)]


>
> I get nothing. And yet I’ve been able to return the variable itself from the
> module and check its type, and use it in a let assignment. So I can see that
> the module is receiving the value I send, it just isn’t matching for some
> reason.
>
> Bryan Cholfin
>
> Associate Editor for Digital Projects and Metadata
> Harvard University Press
> 617.496.9116
> 79 Garden Street | Cambridge, MA 02138-1400
> www.hup.harvard.edu
>
>
> From: Etanchaud Fabrice 
> Date: Tuesday, November 17, 2015 at 4:08 AM
> To: "Cholfin, Bryan" ,
> "basex-talk@mailman.uni-konstanz.de" 
> Subject: RE: [basex-talk] sending variables to xquery module via client api
> (python)
>
> Hi Bryan,
>
>
>
> Could http://docs.basex.org/wiki/Options#BINDINGSgive you a solution ?
>
>
>
> Best regards,
>
> Fabrice
>
>
>
> De : basex-talk-boun...@mailman.uni-konstanz.de
> [mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Cholfin,
> Bryan
> Envoyé : lundi 16 novembre 2015 23:08
> À : basex-talk@mailman.uni-konstanz.de
> Objet : [basex-talk] sending variables to xquery module via client api
> (python)
>
>
>
> Hi—
>
>
>
> Hoping someone can help me with this. I’m trying to write a python script
> that calls an xquery module in the database. It seems to be pretty easy to
> use a line such as:
>
>
>
> session.execute(“run gettext.xqm”)
>
>
>
> and get back the response I want. What I want to do is pass a bound variable
> to the module. The client api has a provision to bind a variable to a query
> object, but I’m not making a query object. I’m simply executing the stored
> query.  I don’t find anything in the documentation to suggest that a
> variable can be added to the run command this way.
>
>
>
> Bryan Cholfin
>
> Associate Editor for Digital Projects and Metadata
> Harvard University Press
> 617.496.9116
> 79 Garden Street | Cambridge, MA 02138-1400
> www.hup.harvard.edu

Re: [basex-talk] sending variables to xquery module via client api (python)

[Etanchaud Fabrice]

Hi Bryan,

Could http://docs.basex.org/wiki/Options#BINDINGS give you a solution ?

Best regards,
Fabrice

De : basex-talk-boun...@mailman.uni-konstanz.de 
[mailto:basex-talk-boun...@mailman.uni-konstanz.de] De la part de Cholfin, Bryan
Envoyé : lundi 16 novembre 2015 23:08
À : basex-talk@mailman.uni-konstanz.de
Objet : [basex-talk] sending variables to xquery module via client api (python)

Hi-

Hoping someone can help me with this. I'm trying to write a python script that 
calls an xquery module in the database. It seems to be pretty easy to use a 
line such as:

session.execute("run gettext.xqm")

and get back the response I want. What I want to do is pass a bound variable to 
the module. The client api has a provision to bind a variable to a query 
object, but I'm not making a query object. I'm simply executing the stored 
query.  I don't find anything in the documentation to suggest that a variable 
can be added to the run command this way.

Bryan Cholfin

Associate Editor for Digital Projects and Metadata
Harvard University Press
617.496.9116
79 Garden Street | Cambridge, MA 02138-1400
www.hup.harvard.edu