Re: [basex-talk] Database folders with numbers after

[James Ball]

So this happened again overnight but I have more information now.

1) I had switched off UPDINDEX on some of the databases and one of these was 
affected so this is not an UPDINDEX issue
2) I found that there was a hard crash of the JVM - which would account for why 
many databases were left locked and one had the folder with the numbers after
3) My Upstart script restarts the server quite happily which disguises 2) if 
you’re not looking closely!

The hard crash was due to the JVM not being able to allocate memory. What I’m 
not sure is why my script to OPTIMIZE all the databases seems to run quite 
happily manually but is failing when run by cron overnight. 

I will continue monitoring.

Regards, James

> On 8 Jan 2016, at 19:11, James Ball  wrote:
> 
> Christian,
> 
> I can’t reliably reproduce the behaviour at all at the moment but I’m losing 
> a fair number of databases to the problem. I first thought it was related to 
> running out of disk space mid-operation but I don’t think that’s the cause. I 
> just wanted to check that I wasn’t doing anything I shouldn’t before I try to 
> find a reproducible error.
> 
> UPDINDEX is really useful for my use case but seems to leave me with database 
> in GB range until I OPTIMIZE them. I shall continue monitoring.
> 
> Many thanks, James
> 
>> On 8 Jan 2016, at 19:00, Christian Grün  wrote:
>> 
>>> So another question - these seem to come about from running OPTIMIZE ALL on 
>>> databases with UPDINDEX TRUE. Is there any reason that OPTIMIZE ALL can’t 
>>> be run on such databases?
>> 
>> It shouldn’t cause any problems. Here is a little command scripts that
>> demonstrates that it should work alright:
>> 
>> 
>>   true
>>   
>>   
>>   
>>   
>>   
>>   
>> 
>> 
>> Can you also reproduce the behavior with smaller data sets?
>> 
>> Thanks,
>> Christian
> 

Re: [basex-talk] Lookups and arrows

[Joe Wicentowski]

Hi Tim,
For your first question, I think your example falls into what the spec calls 
"funky looking" keys.  See the 3rd bullet point example under 
http://www.w3.org/TR/xquery-31/#id-lookup:
> funky /  an appropriate lookup for a map with rather odd conventions for keys.
In other words, I think you're stuck with the ("@context") approach.
As to your second question, it looks like what you're proposing should work - 
but am I reading you as saying you get an error with your proposed approach?  
If so, what's the error?
Joe

Sent from my iPad

_
From: Tim Thompson 
Sent: Saturday, January 9, 2016 3:11 PM
Subject: [basex-talk] Lookups and arrows
To: BaseX 


   
Hello,

   I'm testing some XQuery 3.1 features against a JSON-LD[1] document 
and had a few questions. In the JSON-LD format, the "@" symbol has special 
semantics in key names, but seems to cause problems with the 3.1 lookup 
operator.   
   
  For example:  
  
json-doc("  http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") 
 
  
 works as expected, but 
 
json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)?@context 

 
throws an error: [XPST0003] No specifier after lookup operator: '@'.


   Also, when using the "=>" operator, should it be possible to perform a 
lookup on the last expression in a chain, if that expression returns a map? 
  
   
  For example:  
  
json-doc("  http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") 
evaluates to "  http://iiif.io/api/presentation/2/context.json;  
  
and  
  
json-doc("  http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") => 
json-doc()  
  
returns another map object. So, how would one achieve this: 
 
  
json-doc(json-doc("  
http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context"))("@context")  
  
using the arrow operator?  
  
Thanks, 
 Tim  
 
  
[1]http://www.w3.org/TR/json-ld/   
   
   
  --  
Tim A. Thompson  
Metadata Librarian (Spanish/Portuguese Specialty)  
Princeton University Library  
  
   


  

Re: [basex-talk] group-by behaviour for clustering XML fragments

[Christian Grün]

Hi Constantine,

> Incidentally, BaseX is simply unbelievably fast at executing this – a
> million fragments clustered and written out to another DB in 16 seconds on a
> laptop. My congratulations on an amazing product.

Thanks!

> If I use the entire XML fragment as a grouping key, something like this:
> [...]
> … then will the grouping be equivalent to the functionality of the
> deep-equal function?

A grouping keys is the atomized value of a grouping variable [1,2]. If
this value is prone to be ambiguous, you can create an arbitrary other
value, e.g. as follows:

  group by $val := string-join($a/*, '; ')

Cheers,
Christian

[1] http://docs.basex.org/wiki/XQuery_3.0#group_by
[2] http://www.w3.org/TR/xquery-30/#id-group-by

Re: [basex-talk] Mimicking RDBMS uniqueness constraints

[Christian Grün]

> We are already using UUIDs for file names.  However we want, for
> example, guaranteed uniqueness across text names contained in elements
> within documents of a collection.

I am not sure how "text names contained in elements" will be
represented in your data. If it’s stored in text or attribute nodes,
just try the query I’ve sent.


>
>> On Jan 9, 2016, at 12:52 PM, "Christian Grün"  
>> wrote:
>>
>> Hi E. Wray,
>>
>>> What is the best practice in BaseX to implement a uniqueness
>>> constraint similar to a RDBMS?
>>
>> Using UUIDs has become our favorite solution. An alternative is to
>> assign an incrementing integer, and e.g. store the highest assigned
>> value in the root node. If you want to assign arbitrary values, you’ll
>> need to check if the value has already been assigned:
>>
>>  let $id := 'sjdhsj'
>>  return if(//item[@id = $id])) then (
>>error( (), "ID is not unique: " || $id)
>>  ) else ( ... )
>>
>> If your database has an up-to-date attribute index, this should be pretty 
>> fast.
>>
>> Christian

[basex-talk] Lookups and arrows

[Tim Thompson]

Hello,

I'm testing some XQuery 3.1 features against a JSON-LD[1] document and had
a few questions. In the JSON-LD format, the "@" symbol has special
semantics in key names, but seems to cause problems with the 3.1 lookup
operator.

For example:

json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")

works as expected, but

json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld;)?@context

throws an error: [XPST0003] No specifier after lookup operator: '@'.

Also, when using the "=>" operator, should it be possible to perform a
lookup on the last expression in a chain, if that expression returns a map?

For example:

json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")
evaluates to "http://iiif.io/api/presentation/2/context.json;

and

json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") =>
json-doc()

returns another map object. So, how would one achieve this:

json-doc(json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld
")("@context"))("@context")

using the arrow operator?

Thanks,
Tim


[1] http://www.w3.org/TR/json-ld/


--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

Re: [basex-talk] Lookups and arrows

[Tim Thompson]

Thanks, Joe! I guess I'd glossed over the "funky" example ;-) Regarding the
arrow operator, I was wondering whether something like this was possible:

json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") =>
json-doc()("@context")

which throws an error: [XPST0003] Unexpected end of query: '("@context")'.

Tim

On Sat, Jan 9, 2016 at 3:53 PM, Joe Wicentowski  wrote:

> Hi Tim,
>
> For your first question, I think your example falls into what the spec
> calls "funky looking" keys.  See the 3rd bullet point example under
> http://www.w3.org/TR/xquery-31/#id-lookup:
>
> > funky /  @string"), an appropriate lookup for a map with rather odd conventions for
> keys.
>
> In other words, I think you're stuck with the ("@context") approach.
>
> As to your second question, it looks like what you're proposing should
> work - but am I reading you as saying you get an error with your proposed
> approach?  If so, what's the error?
>
> Joe
>
> Sent from my iPad
>
> _
> From: Tim Thompson 
> Sent: Saturday, January 9, 2016 3:11 PM
> Subject: [basex-talk] Lookups and arrows
> To: BaseX 
>
>
>
> Hello,
>
> I'm testing some XQuery 3.1 features against a JSON-LD[1] document and had
> a few questions. In the JSON-LD format, the "@" symbol has special
> semantics in key names, but seems to cause problems with the 3.1 lookup
> operator.
>
> For example:
>
> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")
>
> works as expected, but
>
> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)?@context
>
> throws an error: [XPST0003] No specifier after lookup operator: '@'.
>
> Also, when using the "=>" operator, should it be possible to perform a
> lookup on the last expression in a chain, if that expression returns a map?
>
> For example:
>
> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")
> evaluates to " http://iiif.io/api/presentation/2/context.json;
>
> and
>
> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") =>
> json-doc()
>
> returns another map object. So, how would one achieve this:
>
> json-doc(json-doc(" 
> http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context"))("@context")
>
>
> using the arrow operator?
>
> Thanks,
> Tim
>
>
> [1] http://www.w3.org/TR/json-ld/
>
>
> --
> Tim A. Thompson
> Metadata Librarian (Spanish/Portuguese Specialty)
> Princeton University Library
>
>
>
>

Re: [basex-talk] Lookups and arrows

[Joe Wicentowski]

Hi Tim,

"Funky" made me chuckle too!

I tested your code with the closest thing at hand - a local build of
branch of Wolfgang Meier's eXist repo with support for the arrow
operator (https://github.com/wolfgangmm/exist/tree/feature/arrowop) -
and your code worked fine there.  So my hunch is that there's a bug in
BaseX or some syntax error I'm not seeing in your code.  For the
record, here's the code I'm running:

--
xquery version "3.1";

json-doc("http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")
=> json-doc()("@context")
--

To check the results I actually use `=>
serialize(http://www.w3.org/2010/xslt-xquery-serialization;>jsonyes)`
on the results since eXist doesn't support adaptive serialization.

I hope this is helpful.

Joe

On Sat, Jan 9, 2016 at 6:20 PM, Tim Thompson  wrote:
> Thanks, Joe! I guess I'd glossed over the "funky" example ;-) Regarding the
> arrow operator, I was wondering whether something like this was possible:
>
> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") =>
> json-doc()("@context")
>
> which throws an error: [XPST0003] Unexpected end of query: '("@context")'.
>
> Tim
>
> On Sat, Jan 9, 2016 at 3:53 PM, Joe Wicentowski  wrote:
>>
>> Hi Tim,
>>
>> For your first question, I think your example falls into what the spec
>> calls "funky looking" keys.  See the 3rd bullet point example under
>> http://www.w3.org/TR/xquery-31/#id-lookup:
>>
>> > funky / > > @string"), an appropriate lookup for a map with rather odd conventions for
>> > keys.
>>
>> In other words, I think you're stuck with the ("@context") approach.
>>
>> As to your second question, it looks like what you're proposing should
>> work - but am I reading you as saying you get an error with your proposed
>> approach?  If so, what's the error?
>>
>> Joe
>>
>> Sent from my iPad
>>
>> _
>> From: Tim Thompson 
>> Sent: Saturday, January 9, 2016 3:11 PM
>> Subject: [basex-talk] Lookups and arrows
>> To: BaseX 
>>
>>
>>
>> Hello,
>>
>> I'm testing some XQuery 3.1 features against a JSON-LD[1] document and had
>> a few questions. In the JSON-LD format, the "@" symbol has special semantics
>> in key names, but seems to cause problems with the 3.1 lookup operator.
>>
>> For example:
>>
>> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")
>>
>> works as expected, but
>>
>> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)?@context
>>
>> throws an error: [XPST0003] No specifier after lookup operator: '@'.
>>
>> Also, when using the "=>" operator, should it be possible to perform a
>> lookup on the last expression in a chain, if that expression returns a map?
>>
>> For example:
>>
>> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context")
>> evaluates to " http://iiif.io/api/presentation/2/context.json;
>>
>> and
>>
>> json-doc(" http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context") =>
>> json-doc()
>>
>> returns another map object. So, how would one achieve this:
>>
>> json-doc(json-doc("
>> http://lae.princeton.edu/catalog/0bp35.jsonld;)("@context"))("@context")
>>
>> using the arrow operator?
>>
>> Thanks,
>> Tim
>>
>>
>> [1] http://www.w3.org/TR/json-ld/
>>
>>
>> --
>> Tim A. Thompson
>> Metadata Librarian (Spanish/Portuguese Specialty)
>> Princeton University Library
>>
>>
>>
>