Hi, I am also interested in seeing the patch for this, thanks! --Hardy
________________________________ From: [email protected] [[email protected]] on behalf of [email protected] [[email protected]] Sent: Tuesday, March 22, 2016 1:50 PM To: DSpace Technical Support Cc: [email protected] Subject: Re: [dspace-tech] Re: sort order in subject browse/facet - diacritics Andrea, It's working for us for the browse index too: https://papyrus.bib.umontreal.ca/xmlui/handle/1866/1319/browse?rpp=20&sort_by=-1&type=subject&offset=1350&etal=-1&order=ASC What is not working though (I just realised this) is that if you try to enter the first letters of a word in order to position the searched term within the index, eg. "économie", then I do not get the same result (position), as when I am typing "economie" (without accent). We will work on that. Le lundi 21 mars 2016 20:38:03 UTC-4, Andrea Schweer a écrit : Hi Marie-Hélène, your approach sounds very promising, thank you! It'd be great if you could share your patch. Though I suspect this will only work for facets, not for the browse (since I believe the browse doesn't store the value in that lower-case/real value form). cheers, Andrea On 19/03/16 08:51, [email protected]<UrlBlockedError.aspx> wrote: Hi Andrea, Since the facets are already modified in order to have lower case values, followed by a separator, followed by their real value, ex.: "aboriginal rights\n|||\naboriginal rights",1, "abricot\n|||\nabricot",3, "access to care\n|||\naccess to care",1, "access to healthcare\n|||\naccess to healthcare",1, "accessibilité aux soins\n|||\naccessibilité aux soins",1, "accountability\n|||\naccountability",1, "accès aux soins de santé\n|||\naccès aux soins de santé",1, "actes de colloque\n|||\nactes de colloque",1, "activité motrice\n|||\nActivité motrice",1, We have altered the first form (that is before the separator), in order to also have the accented characters replaced by their unaccented counterpart. Only the form used for sorting is changed, not the form used for display. For that we are using the Apache commons "stripAccents" (https://commons.apache.org/proper/commons-lang/javadocs/api-3.1/org/apache/commons/lang3/StringUtils.html) <https://commons.apache.org/proper/commons-lang/javadocs/api-3.1/org/apache/commons/lang3/StringUtils.html%29%C2%A0> which removes diacritics from a string without altering the case. We have added the following dependency in [SOURCES]/dspace-api/pom.xml: <dependency> <groupId>commons-lang</groupId> <artifactId>commons-lang3</artifactId> <version>3.4</version> </dependency> And we have changed all occurrences of OrderFormat within the [SOURCES]/dspace-api/src/main/java/org/dspace/browse/SolrBrowseCreateDAO.java for org.apache.commons.lang3.StringUtils.stripAccents(OrderFormat (...)). And within [SOURCES]/dspace-api/src/main/java/org/dspace/discovery/SolrServiceImpl.java where values are set to lower case (a few occurrences), we have added the stripAccents, which will give something similar to: org.apache.commons.lang3.StringUtils.stripAccents(value.toLowerCase()) org.apache.commons.lang3.StringUtils.stripAccents(facetValue.toLowerCase()) org.apache.commons.lang3.StringUtils.stripAccents(indexValue.toLowerCase()) etc.... We did reindex and now we have: "aboriginal rights\n|||\naboriginal rights",1, "abricot\n|||\nabricot",3, "acces aux soins de sante\n|||\naccès aux soins de santé",1, "access to care\n|||\naccess to care",1, "access to healthcare\n|||\naccess to healthcare",1, "accessibilite aux soins\n|||\naccessibilité aux soins",1, "accountability\n|||\naccountability",1, "actes de colloque\n|||\nactes de colloque",1, "activite motrice\n|||\nActivité motrice",1, The resulting are indexes where diacritics are not taken into account for sort order (https://papyrus.bib.umontreal.ca/xmlui/handle/1866/1319/browse?rpp=20&sort_by=-1&type=subject&offset=1350&etal=-1&order=ASC): Economic, social and cultural rights [1] Economical Europe [1] Économie [1] Économie et droit de l'homme [1] Economie sanction [1] Economy [1] ecosystem approach [1] Ecrit [1] Écrit [2] écrit [1] (...) Is that what you are looking for? I can send you our modified files if you wish. regards, Marie-Hélène Vézina Université de Montréal Le mercredi 24 février 2016 23:18:49 UTC-5, Andrea Schweer a écrit : Hi all, has anyone had any luck getting the subject browse/facets to ignore diacritics when sorting the subject terms? Right now (5.x, XMLUI, discovery), we get a sort order like this: manager Manuka Leaf Oil Manukau Manukau Harbour marginal groups marginal value theorem Maze procedure Mānuka Honey in browse and manager Manuka Leaf Oil Manukau Manukau Harbour marginal groups marginal value theorem Maze procedure mythology Mānuka Honey in the facet listing. I've been asked whether it's possible to change the sort order such that "Mānuka" gets sorted between "manager" and "Manukau" -- that is, as if there was no diacritic. The subject terms should remain listed with their actual values though (it's not an option to simply strip out the diacritics during indexing). I have vague memories that we managed to tweak this behaviour by configuring the collation locale in PostgreSQL, back in the database-backed browse. Obviously that doesn't work with Discovery. I tried a few changes to the Solr schema for the discovery core, but none appeared to do what I want. Based on the UnicodeCollation page in the Solr wiki [1], it looks like configuring collation for a sort field might be the way to go; it isn't possible to set up collation for the subject field/s because then the stored value changes to a binary representation of the metadata value that includes the collation key. I imagine this would work for title browse, where the results (=DSpace items) are sorted by one of the configured sort fields. However, the subject browse listings (where the results are metadata values) ultimately come from a facet query, and facet.sort only takes "count" or "index" as its values [2] -- it doesn't look like it's possible to add collation into the mix there. Does anyone have any thoughts on this? cheers, Andrea [1]: https://wiki.apache.org/solr/UnicodeCollation [2]: https://wiki.apache.org/solr/SimpleFacetParameters#facet.sort -- Dr Andrea Schweer Lead Software Developer, ITS Information Systems The University of Waikato, Hamilton, New Zealand +64-7-837 9120 -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<UrlBlockedError.aspx>. To post to this group, send email to [email protected]<UrlBlockedError.aspx>. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout. -- Dr Andrea Schweer Lead Software Developer, ITS Information Systems The University of Waikato, Hamilton, New Zealand +64-7-837 9120 -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To post to this group, send email to [email protected]<mailto:[email protected]>. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
