Re: [MarkLogic Dev General] Whitespace, Punctuation, Collations, and &

Thu, 25 Jun 2009 09:53:23 -0700 


The issue here isn't the lexicon and its collation, the issue here is
the element-value-query.

The query used with a a lexicon operation in this way is executed
unfiltered, and a non-exact (i.e. whitespace-insensitive)
punctuation-sensitive value query does not resolve the difference
between "Personality & Individ. Diff" and
"Personality Individ. Diff" in the index.

What you need to do is make sure that you only use queries that
are accurate against the index (i.e. where fn:count=xdmp:estimate).

//Mary

On Thu, 25 Jun 2009 09:08:09 -0700, Tim Meagher <[email protected]> wrote:


Hi Folks,
I have come across an interesting phenomenon and am trying to understand it. I have a (range element index) lexicon configured for JournalTitle using the 
root collation which contains the following distinct values:


Personality & Individ. Diff.


and


Personality Individ. Diff.
If I perform a lexicon search for "Personality &amp; Individ. Diff." using cts:element-values() with a subquery of cts:element-value-query() specifying 
the equivalent of an exact match in the search options as follows:



element results {

  for $result in

    cts:element-values(xs:QName("JournalTitle"), (),

       ("item-frequency", "item-order", "ascending"),

       cts:and-query((

         cts:element-value-query(xs:QName("JournalTitle"),

                 "Personality &amp; Individ. Diff.",

           ("case-sensitive", "diacritic-sensitive",

            "punctuation-sensitive", "whitespace-sensitive",

             "unstemmed", "unwildcarded"))

       ))

     )[1 to 20]

  return element result {$result}

}


then I get the following results (as expected):


 <http://markprod.apa.org:8002/eval.xqy?iefix.txt##> - <results>

  <result>Personality & Individ. Diff.</result>

     </results>
However, if I change the request options to whitespace-insensitive, then I 
get the following results:


 <http://markprod.apa.org:8002/eval.xqy?iefix.txt##> - <results>

  <result>Personality & Individ. Diff.</result>

  <result>Personality Individ. Diff.</result>

     </results>
This implies to me that the ampersand is treated like whitespace. I would have expected it to be treated as punctuation, but I'm not sure exactly what character set (including escaped characters) are defined by whitespace and 
punctuation.  I've looked into the UCA and ISO-8859-1 specs to try to
understand the default MarkLogic root collation, but I haven't found a
simple list that would help me to understand why I'm getting the above
results.  Can anyone shed some light on this?
Can someone also help clarify the distinction between the default MarkLogic root collation (http://marklogic.com/collation) and the codepoint collation 
(http://marklogic.com/collation/codepoint)?  I'm trying to find the ideal
collation for my JournalTitle lexicon collation.


Thanks for the help!


Tim Meagher - AAOM Consulting





_______________________________________________
General mailing list
[email protected]
http://xqzone.com/mailman/listinfo/general

Reply via email to