Re: [Dspace-tech] search can't sort by date issued
Hi Peter, Do you know if this been fixed in the latest version? We are currently running 1.7.2 and have this problem with date issued metadata values in the multiple formats you have mentioned. In the latest version (1.8.1) it looks like as part of the lucene upgrade (DS-980) the sort fields are now explicitly specified as STRING types: http://scm.dspace.org/svn/repo/dspace/tags/dspace-1.8.1/dspace-api/src/main/java/org/dspace/search/DSQuery.java (~line 225) new SortField(sort_ + args.getSortOption().getName(), SortField.STRING, SortOption.DESCENDING.equals(args.getSortOrder())), SortField.FIELD_SCORE The second parameter (SortField.STRING) is not used in 1.7.2 but could be and editing this line seems to resolve the issue. I'm not that familiar with the indexing - could this be a temporary fix until we upgrade or have I missed something? Regards Pete From: Peter Dietz [mailto:pdiet...@gmail.com] Sent: 03 February 2012 20:01 To: Thornton, Susan M. (LARC-B702)[LITES] Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] search can't sort by date issued I'll also add that we have two repositories, and one of them can you can search, and it will sort properly. And the other has the problems identified in this thread. I'm sure both repositories has equally screwy metadata for dates. So I'm not entirely sure this is going to be the end of it. Peter Dietz On Fri, Feb 3, 2012 at 2:51 PM, Peter Dietz pdiet...@gmail.com wrote: Hi Sue, I've been diagnosing this issue in Luke (a Java GUI that allows you browse your lucene index). And while digging around, it looked like the sort_dateissued field is having trouble with certain date metadata. In our repository, we have our date metadata values scattered all about. 1981-12-07T16:56:12Z 1981-12-07 1981-12 1981 Each one of them is a valid ISO8601 date. However, that doesn't mean each of them is a valid date in Lucene (your search and browse index). A metadata person might see 1981-12 as meaning some type of range or approximation. However, when you are searching and sorting, it must be able to sort the values precisely. So is 1981-12 before or after 1981-12-07? Is 1981-12 before, after, or equal to 1981-12-01? I'll ask my metadata people if we can flatten our metadata for dates, and stuff them to have a day (of the first of the month). And I'll dig further into the DSpace reindexing code to see if when we are processing DSpace metadata dates, that might be valid iso8601, that we convert them to an appropriate lucene date. Peter Dietz On Thu, Feb 2, 2012 at 11:55 PM, Thornton, Susan M. (LARC-B702)[LITES] susan.m.thorn...@nasa.gov wrote: Thanks Peter. I've spent several hours researching this issue, especially why we have it in one DSpace instance and not another (running same versions). Although I'm not 100% sure, I suspect the issue is caused by invalid data in the date.issued field(s) in the respository. The solution for this, of course, would be to clean up the bad dates we have and then put some edits on the date fields that end up in DSpace so we do not allow bad dates to get IN our repository. But again, I'm not 100% sure of this and I won't be able to get back to looking into this for awhile. Best regards, Sue Sue Walker-Thornton Software Developer|Database Administrator NASA Langley Research Center SGT, Inc.|LITES Contract 130 Research Drive Hampton, VA 23666 Office: (757) 864-2368|Fax: (757) 224-4001|Mobile: (757) 506-9903 Email: susan.m.thorn...@nasa.gov From: Peter Dietz [pdiet...@gmail.com] Sent: Wednesday, February 01, 2012 12:29 PM To: Cristian Romanescu Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] search can't sort by date issued Hi All, I've just started digging into this as well. Its really unfortunate to only get relevance results for searches. In digging in, I've spit out the stack trace, and its telling me a few things. 1) Do we have bad metadata for dc.date.issued? -- (I've already harassed my content folks to have them review all our metadata) ;) 2) Are we doing the comparison of dates incorrectly. The error below says is the value of dateissued an INT. -- I've been reading this thread, which is very similar: http://www.gossamer-threads.com/lists/lucene/java-user/109530 2012-01-31 17:47:02,475 ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued 2012-01-31 17:47:02,475 ERROR org.dspace.search.DSQuery @ Invalid shift value in prefixCoded string (is encoded value really an INT?) 2012-01-31 17:47:02,476 ERROR org.dspace.search.DSQuery @ java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really an INT?) at org.apache.lucene.util.NumericUtils.prefixCodedToInt(NumericUtils.java:233) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:237) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java
Re: [Dspace-tech] search can't sort by date issued
Hi Sue, I've been diagnosing this issue in Luke (a Java GUI that allows you browse your lucene index). And while digging around, it looked like the sort_dateissued field is having trouble with certain date metadata. In our repository, we have our date metadata values scattered all about. 1981-12-07T16:56:12Z 1981-12-07 1981-12 1981 Each one of them is a valid ISO8601 date. However, that doesn't mean each of them is a valid date in Lucene (your search and browse index). A metadata person might see 1981-12 as meaning some type of range or approximation. However, when you are searching and sorting, it must be able to sort the values precisely. So is 1981-12 before or after 1981-12-07? Is 1981-12 before, after, or equal to 1981-12-01? I'll ask my metadata people if we can flatten our metadata for dates, and stuff them to have a day (of the first of the month). And I'll dig further into the DSpace reindexing code to see if when we are processing DSpace metadata dates, that might be valid iso8601, that we convert them to an appropriate lucene date. Peter Dietz On Thu, Feb 2, 2012 at 11:55 PM, Thornton, Susan M. (LARC-B702)[LITES] susan.m.thorn...@nasa.gov wrote: Thanks Peter. I've spent several hours researching this issue, especially why we have it in one DSpace instance and not another (running same versions). Although I'm not 100% sure, I suspect the issue is caused by invalid data in the date.issued field(s) in the respository. The solution for this, of course, would be to clean up the bad dates we have and then put some edits on the date fields that end up in DSpace so we do not allow bad dates to get IN our repository. But again, I'm not 100% sure of this and I won't be able to get back to looking into this for awhile. Best regards, Sue Sue Walker-Thornton Software Developer|Database Administrator NASA Langley Research Center SGT, Inc.|LITES Contract 130 Research Drive Hampton, VA 23666 Office: (757) 864-2368|Fax: (757) 224-4001|Mobile: (757) 506-9903 Email: susan.m.thorn...@nasa.gov -- *From:* Peter Dietz [pdiet...@gmail.com] *Sent:* Wednesday, February 01, 2012 12:29 PM *To:* Cristian Romanescu *Cc:* dspace-tech@lists.sourceforge.net *Subject:* Re: [Dspace-tech] search can't sort by date issued Hi All, I've just started digging into this as well. Its really unfortunate to only get relevance results for searches. In digging in, I've spit out the stack trace, and its telling me a few things. 1) Do we have bad metadata for dc.date.issued? -- (I've already harassed my content folks to have them review all our metadata) ;) 2) Are we doing the comparison of dates incorrectly. The error below says is the value of dateissued an INT. -- I've been reading this thread, which is very similar: http://www.gossamer-threads.com/lists/lucene/java-user/109530 2012-01-31 17:47:02,475 ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued 2012-01-31 17:47:02,475 ERROR org.dspace.search.DSQuery @ Invalid shift value in prefixCoded string (is encoded value really an INT?) 2012-01-31 17:47:02,476 ERROR org.dspace.search.DSQuery @ java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really an INT?) at org.apache.lucene.util.NumericUtils.prefixCodedToInt(NumericUtils.java:233) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:237) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:457) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:447) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$MultiComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:435) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:249) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:240) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:181) at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:113) at org.apache.lucene.search.Hits.init(Hits.java:90) at org.apache.lucene.search.Searcher.search(Searcher.java:63) at org.dspace.search.DSQuery.doQuery(DSQuery.java:151) at org.dspace.search.DSQuery.doQuery(DSQuery.java:309) at org.dspace.app.xmlui.aspect.artifactbrowser.AbstractSearch.performSearch(AbstractSearch.java:438) Just for fun, I enabled Discovery on our development machines, and sorting by date issued works perfectly in a search. So, a quick-fix would be to switch to using discovery. But, none-the-less, I look
Re: [Dspace-tech] search can't sort by date issued
I'll also add that we have two repositories, and one of them can you can search, and it will sort properly. And the other has the problems identified in this thread. I'm sure both repositories has equally screwy metadata for dates. So I'm not entirely sure this is going to be the end of it. Peter Dietz On Fri, Feb 3, 2012 at 2:51 PM, Peter Dietz pdiet...@gmail.com wrote: Hi Sue, I've been diagnosing this issue in Luke (a Java GUI that allows you browse your lucene index). And while digging around, it looked like the sort_dateissued field is having trouble with certain date metadata. In our repository, we have our date metadata values scattered all about. 1981-12-07T16:56:12Z 1981-12-07 1981-12 1981 Each one of them is a valid ISO8601 date. However, that doesn't mean each of them is a valid date in Lucene (your search and browse index). A metadata person might see 1981-12 as meaning some type of range or approximation. However, when you are searching and sorting, it must be able to sort the values precisely. So is 1981-12 before or after 1981-12-07? Is 1981-12 before, after, or equal to 1981-12-01? I'll ask my metadata people if we can flatten our metadata for dates, and stuff them to have a day (of the first of the month). And I'll dig further into the DSpace reindexing code to see if when we are processing DSpace metadata dates, that might be valid iso8601, that we convert them to an appropriate lucene date. Peter Dietz On Thu, Feb 2, 2012 at 11:55 PM, Thornton, Susan M. (LARC-B702)[LITES] susan.m.thorn...@nasa.gov wrote: Thanks Peter. I've spent several hours researching this issue, especially why we have it in one DSpace instance and not another (running same versions). Although I'm not 100% sure, I suspect the issue is caused by invalid data in the date.issued field(s) in the respository. The solution for this, of course, would be to clean up the bad dates we have and then put some edits on the date fields that end up in DSpace so we do not allow bad dates to get IN our repository. But again, I'm not 100% sure of this and I won't be able to get back to looking into this for awhile. Best regards, Sue Sue Walker-Thornton Software Developer|Database Administrator NASA Langley Research Center SGT, Inc.|LITES Contract 130 Research Drive Hampton, VA 23666 Office: (757) 864-2368|Fax: (757) 224-4001|Mobile: (757) 506-9903 Email: susan.m.thorn...@nasa.gov -- *From:* Peter Dietz [pdiet...@gmail.com] *Sent:* Wednesday, February 01, 2012 12:29 PM *To:* Cristian Romanescu *Cc:* dspace-tech@lists.sourceforge.net *Subject:* Re: [Dspace-tech] search can't sort by date issued Hi All, I've just started digging into this as well. Its really unfortunate to only get relevance results for searches. In digging in, I've spit out the stack trace, and its telling me a few things. 1) Do we have bad metadata for dc.date.issued? -- (I've already harassed my content folks to have them review all our metadata) ;) 2) Are we doing the comparison of dates incorrectly. The error below says is the value of dateissued an INT. -- I've been reading this thread, which is very similar: http://www.gossamer-threads.com/lists/lucene/java-user/109530 2012-01-31 17:47:02,475 ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued 2012-01-31 17:47:02,475 ERROR org.dspace.search.DSQuery @ Invalid shift value in prefixCoded string (is encoded value really an INT?) 2012-01-31 17:47:02,476 ERROR org.dspace.search.DSQuery @ java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really an INT?) at org.apache.lucene.util.NumericUtils.prefixCodedToInt(NumericUtils.java:233) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:237) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:457) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:447) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$MultiComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:435) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:249) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:240) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:181) at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:113) at org.apache.lucene.search.Hits.init(Hits.java:90) at org.apache.lucene.search.Searcher.search(Searcher.java:63
Re: [Dspace-tech] search can't sort by date issued
Thanks Peter. I've spent several hours researching this issue, especially why we have it in one DSpace instance and not another (running same versions). Although I'm not 100% sure, I suspect the issue is caused by invalid data in the date.issued field(s) in the respository. The solution for this, of course, would be to clean up the bad dates we have and then put some edits on the date fields that end up in DSpace so we do not allow bad dates to get IN our repository. But again, I'm not 100% sure of this and I won't be able to get back to looking into this for awhile. Best regards, Sue Sue Walker-Thornton Software Developer|Database Administrator NASA Langley Research Center SGT, Inc.|LITES Contract 130 Research Drive Hampton, VA 23666 Office: (757) 864-2368|Fax: (757) 224-4001|Mobile: (757) 506-9903 Email: susan.m.thorn...@nasa.gov From: Peter Dietz [pdiet...@gmail.com] Sent: Wednesday, February 01, 2012 12:29 PM To: Cristian Romanescu Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] search can't sort by date issued Hi All, I've just started digging into this as well. Its really unfortunate to only get relevance results for searches. In digging in, I've spit out the stack trace, and its telling me a few things. 1) Do we have bad metadata for dc.date.issued? -- (I've already harassed my content folks to have them review all our metadata) ;) 2) Are we doing the comparison of dates incorrectly. The error below says is the value of dateissued an INT. -- I've been reading this thread, which is very similar: http://www.gossamer-threads.com/lists/lucene/java-user/109530 2012-01-31 17tel:2012-01-31%2017:47:02,475 ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued 2012-01-31 17tel:2012-01-31%2017:47:02,475 ERROR org.dspace.search.DSQuery @ Invalid shift value in prefixCoded string (is encoded value really an INT?) 2012-01-31 17tel:2012-01-31%2017:47:02,476 ERROR org.dspace.search.DSQuery @ java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really an INT?) at org.apache.lucene.util.NumericUtils.prefixCodedToInt(NumericUtils.java:233) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:237) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:457) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:447) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$MultiComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:435) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:249) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:240) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:181) at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:113) at org.apache.lucene.search.Hits.init(Hits.java:90) at org.apache.lucene.search.Searcher.search(Searcher.java:63) at org.dspace.search.DSQuery.doQuery(DSQuery.java:151) at org.dspace.search.DSQuery.doQuery(DSQuery.java:309) at org.dspace.app.xmlui.aspect.artifactbrowser.AbstractSearch.performSearch(AbstractSearch.java:438) Just for fun, I enabled Discovery on our development machines, and sorting by date issued works perfectly in a search. So, a quick-fix would be to switch to using discovery. But, none-the-less, I look forward to getting a resolution to this issue. Peter Dietz On Wed, Feb 1, 2012 at 7:15 AM, Cristian Romanescu cristian.romane...@eaudeweb.romailto:cristian.romane...@eaudeweb.ro wrote: Greetings, Have you tried to look into the lucene indexes with Luke tool? (http://www.getopt.org/luke/). We are using: search.index.13 = dc_date:dc.date.issued:date to filter by time interval and it works. But first, we had to remove the old indexes and re-create them to have correct indexing (ie rm -rf $builddir/search and run ./$builddir/bin/dspace index-init). It only worked when data inside index looks like 20120101 ... - when you look with luke tool HTH, Cristian On 02/01/2012 12:46 PM, Päivi Rosenström wrote: Any solution for this found yet ? Thanks! Päivi Re: [Dspace-tech] search can't sort by date issued From: James Bardinjbardin@bu... - 2011-10-27 19:23 On Thu, Oct 27, 2011 at 1:52 PM, Blanco, Joseblancoj@... wrote: # Browse indexes webui.browse.index.1 = title:item:title webui.browse.index.2 = author:metadata:dc.contributor.author:text webui.browse.index.3 = subject:metadata:dc.subject.*:text webui.browse.index.4 = dateissued:item:dateissued
Re: [Dspace-tech] search can't sort by date issued
Any solution for this found yet ? Thanks! Päivi Re: [Dspace-tech] search can't sort by date issued From: James Bardin jbardin@bu... - 2011-10-27 19:23 On Thu, Oct 27, 2011 at 1:52 PM, Blanco, Jose blancoj@... wrote: # Browse indexes webui.browse.index.1 = title:item:title webui.browse.index.2 = author:metadata:dc.contributor.author:text webui.browse.index.3 = subject:metadata:dc.subject.*:text webui.browse.index.4 = dateissued:item:dateissued # Sorting options webui.itemlist.sort-option.1 = title:dc.title:title webui.itemlist.sort-option.2 = dateissued:dc.date.issued:date webui.itemlist.sort-option.3 = dateaccessioned:dc.date.accessioned:date Yeah, I have dateissued in both the browse.index and sort-option, like above. Sorting by dateissued *does* work in browsing, but not for search results (I think search result ordering is done by lucene, and not the webui). I took a guess and added another search index for dateissued:dc.date.issued:date, but that doesn't seem to have any effect. -jim -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
Greetings, Have you tried to look into the lucene indexes with Luke tool? (http://www.getopt.org/luke/). We are using: search.index.13 = dc_date:dc.date.issued:date to filter by time interval and it works. But first, we had to remove the old indexes and re-create them to have correct indexing (ie rm -rf $builddir/search and run ./$builddir/bin/dspace index-init). It only worked when data inside index looks like 20120101 ... - when you look with luke tool HTH, Cristian On 02/01/2012 12:46 PM, Päivi Rosenström wrote: Any solution for this found yet ? Thanks! Päivi Re: [Dspace-tech] search can't sort by date issued From: James Bardinjbardin@bu... - 2011-10-27 19:23 On Thu, Oct 27, 2011 at 1:52 PM, Blanco, Joseblancoj@... wrote: # Browse indexes webui.browse.index.1 = title:item:title webui.browse.index.2 = author:metadata:dc.contributor.author:text webui.browse.index.3 = subject:metadata:dc.subject.*:text webui.browse.index.4 = dateissued:item:dateissued # Sorting options webui.itemlist.sort-option.1 = title:dc.title:title webui.itemlist.sort-option.2 = dateissued:dc.date.issued:date webui.itemlist.sort-option.3 = dateaccessioned:dc.date.accessioned:date Yeah, I have dateissued in both the browse.index and sort-option, like above. Sorting by dateissued *does* work in browsing, but not for search results (I think search result ordering is done by lucene, and not the webui). I took a guess and added another search index for dateissued:dc.date.issued:date, but that doesn't seem to have any effect. -jim -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
Hi All, I've just started digging into this as well. Its really unfortunate to only get relevance results for searches. In digging in, I've spit out the stack trace, and its telling me a few things. 1) Do we have bad metadata for dc.date.issued? -- (I've already harassed my content folks to have them review all our metadata) ;) 2) Are we doing the comparison of dates incorrectly. The error below says is the value of dateissued an INT. -- I've been reading this thread, which is very similar: http://www.gossamer-threads.com/lists/lucene/java-user/109530 2012-01-31 17:47:02,475 ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued 2012-01-31 17:47:02,475 ERROR org.dspace.search.DSQuery @ Invalid shift value in prefixCoded string (is encoded value really an INT?) 2012-01-31 17:47:02,476 ERROR org.dspace.search.DSQuery @ java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really an INT?) at org.apache.lucene.util.NumericUtils.prefixCodedToInt(NumericUtils.java:233) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:237) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:457) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:447) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$MultiComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:435) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:249) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:240) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:181) at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:113) at org.apache.lucene.search.Hits.init(Hits.java:90) at org.apache.lucene.search.Searcher.search(Searcher.java:63) at org.dspace.search.DSQuery.doQuery(DSQuery.java:151) at org.dspace.search.DSQuery.doQuery(DSQuery.java:309) at org.dspace.app.xmlui.aspect.artifactbrowser.AbstractSearch.performSearch(AbstractSearch.java:438) Just for fun, I enabled Discovery on our development machines, and sorting by date issued works perfectly in a search. So, a quick-fix would be to switch to using discovery. But, none-the-less, I look forward to getting a resolution to this issue. Peter Dietz On Wed, Feb 1, 2012 at 7:15 AM, Cristian Romanescu cristian.romane...@eaudeweb.ro wrote: Greetings, Have you tried to look into the lucene indexes with Luke tool? (http://www.getopt.org/luke/). We are using: search.index.13 = dc_date:dc.date.issued:date to filter by time interval and it works. But first, we had to remove the old indexes and re-create them to have correct indexing (ie rm -rf $builddir/search and run ./$builddir/bin/dspace index-init). It only worked when data inside index looks like 20120101 ... - when you look with luke tool HTH, Cristian On 02/01/2012 12:46 PM, Päivi Rosenström wrote: Any solution for this found yet ? Thanks! Päivi Re: [Dspace-tech] search can't sort by date issued From: James Bardinjbardin@bu... - 2011-10-27 19:23 On Thu, Oct 27, 2011 at 1:52 PM, Blanco, Joseblancoj@... wrote: # Browse indexes webui.browse.index.1 = title:item:title webui.browse.index.2 = author:metadata:dc.contributor.author:text webui.browse.index.3 = subject:metadata:dc.subject.*:text webui.browse.index.4 = dateissued:item:dateissued # Sorting options webui.itemlist.sort-option.1 = title:dc.title:title webui.itemlist.sort-option.2 = dateissued:dc.date.issued:date webui.itemlist.sort-option.3 = dateaccessioned:dc.date.accessioned:date Yeah, I have dateissued in both the browse.index and sort-option, like above. Sorting by dateissued *does* work in browsing, but not for search results (I think search result ordering is done by lucene, and not the webui). I took a guess and added another search index for dateissued:dc.date.issued:date, but that doesn't seem to have any effect. -jim -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] search can't sort by date issued
Hello, We have also encountered this problem (1.7.2/xmlui) - did anyone find a solution or could offer any advice? Thanks Pete -Original Message- From: Thornton, Susan M. (LARC-B702)[LITES] susan.m.thorn...@nasa.gov Subject:Re: [Dspace-tech] search can't sort by date issued Date: Mon, 31 Oct 2011 14:27:07 -0500 (31/10/11 19:27:07) I have been struggling with this same problem for the past 2 weeks and have been unable to resolve it as of yet. I would appreciate anyone's help!! We actually had a user report this with the DSpace feedback feature on our site. Thanks a bunch, Sue Sue Walker-Thornton (757) 864-2368 -Original Message- From: James Bardin [mailto:jbar...@bu.edu] Sent: Thursday, October 27, 2011 1:12 PM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] search can't sort by date issued Hello, Attempting to sort search results by issue date results in the following error: ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued Checking DSQuery.java (line ~235) shows that this is catching a generic exception from lucene, which results in the sorting falling back to relevance. This is confusing to users, because there no indication from the xmlui that would indicate the sorting failed or is incorrect. Any ideas why sorting on dc.date.isssued is failing, or where to start for troubleshooting? Thanks, -- James Bardin jbar...@bu.edu Systems Engineer Boston University IST -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
Apologies if this has been asked before, but is this occurring with the regular search or with Discovery enabled? Bram -- [image: logo] *Bram Luyten* *@mire* *2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010* *Esperantolaan 4, Heverlee 3001, Belgium* http://www.atmire.com/www.atmire.com On Tue, Dec 20, 2011 at 11:30 AM, Peter Warrington pwarring...@dmu.ac.ukwrote: Hello, We have also encountered this problem (1.7.2/xmlui) - did anyone find a solution or could offer any advice? Thanks Pete -Original Message- From: Thornton, Susan M. (LARC-B702)[LITES] susan.m.thorn...@nasa.gov Subject:Re: [Dspace-tech] search can't sort by date issued Date: Mon, 31 Oct 2011 14:27:07 -0500 (31/10/11 19:27:07) I have been struggling with this same problem for the past 2 weeks and have been unable to resolve it as of yet. I would appreciate anyone's help!! We actually had a user report this with the DSpace feedback feature on our site. Thanks a bunch, Sue Sue Walker-Thornton (757) 864-2368 -Original Message- From: James Bardin [mailto:jbar...@bu.edu] Sent: Thursday, October 27, 2011 1:12 PM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] search can't sort by date issued Hello, Attempting to sort search results by issue date results in the following error: ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued Checking DSQuery.java (line ~235) shows that this is catching a generic exception from lucene, which results in the sorting falling back to relevance. This is confusing to users, because there no indication from the xmlui that would indicate the sorting failed or is incorrect. Any ideas why sorting on dc.date.isssued is failing, or where to start for troubleshooting? Thanks, -- James Bardin jbar...@bu.edu Systems Engineer Boston University IST -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
For us, this occurs with the regular search. Pete From: bluy...@gmail.com [mailto:bluy...@gmail.com] On Behalf Of Bram Luyten Sent: 20 December 2011 13:16 To: Peter Warrington Cc: dspace-tech@lists.sourceforge.net; susan.m.thorn...@nasa.gov Subject: Re: [Dspace-tech] search can't sort by date issued Apologies if this has been asked before, but is this occurring with the regular search or with Discovery enabled? Bram -- logohttp://atmire.com/images/@mire_web_2.jpg Bram Luyten @mire 2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010 Esperantolaan 4, Heverlee 3001, Belgium http://www.atmire.com/ www.atmire.com http://www.atmire.com/ On Tue, Dec 20, 2011 at 11:30 AM, Peter Warrington pwarring...@dmu.ac.uk wrote: Hello, We have also encountered this problem (1.7.2/xmlui) - did anyone find a solution or could offer any advice? Thanks Pete -Original Message- From: Thornton, Susan M. (LARC-B702)[LITES] susan.m.thorn...@nasa.gov Subject:Re: [Dspace-tech] search can't sort by date issued Date: Mon, 31 Oct 2011 14:27:07 -0500 (31/10/11 19:27:07) I have been struggling with this same problem for the past 2 weeks and have been unable to resolve it as of yet. I would appreciate anyone's help!! We actually had a user report this with the DSpace feedback feature on our site. Thanks a bunch, Sue Sue Walker-Thornton (757) 864-2368 tel:%28757%29%20864-2368 -Original Message- From: James Bardin [mailto:jbar...@bu.edu] Sent: Thursday, October 27, 2011 1:12 PM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] search can't sort by date issued Hello, Attempting to sort search results by issue date results in the following error: ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued Checking DSQuery.java (line ~235) shows that this is catching a generic exception from lucene, which results in the sorting falling back to relevance. This is confusing to users, because there no indication from the xmlui that would indicate the sorting failed or is incorrect. Any ideas why sorting on dc.date.isssued is failing, or where to start for troubleshooting? Thanks, -- James Bardin jbar...@bu.edu Systems Engineer Boston University IST -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
My very wild guess would be that it's related to a value in date issued that breaks the creation of the index on this value. This could be verified if you can setup an almost empty DSpace, with a few records of which you know they all have valid values for date issued. After this test, we would know whether it's related to the metadata or whether there's a deeper problem. -- [image: logo] *Bram Luyten* *@mire* *2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010* *Esperantolaan 4, Heverlee 3001, Belgium* http://www.atmire.com/www.atmire.com On Tue, Dec 20, 2011 at 2:26 PM, Peter Warrington pwarring...@dmu.ac.ukwrote: For us, this occurs with the regular search. ** ** Pete ** ** *From:* bluy...@gmail.com [mailto:bluy...@gmail.com] *On Behalf Of *Bram Luyten *Sent:* 20 December 2011 13:16 *To:* Peter Warrington *Cc:* dspace-tech@lists.sourceforge.net; susan.m.thorn...@nasa.gov *Subject:* Re: [Dspace-tech] search can't sort by date issued ** ** Apologies if this has been asked before, but is this occurring with the regular search or with Discovery enabled? Bram -- [image: logo] *Bram Luyten* *@mire* *2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010* *Esperantolaan 4, Heverlee 3001, Belgium* http://www.atmire.com/www.atmire.com On Tue, Dec 20, 2011 at 11:30 AM, Peter Warrington pwarring...@dmu.ac.uk wrote: Hello, We have also encountered this problem (1.7.2/xmlui) - did anyone find a solution or could offer any advice? Thanks Pete -Original Message- From: Thornton, Susan M. (LARC-B702)[LITES] susan.m.thorn...@nasa.gov Subject:Re: [Dspace-tech] search can't sort by date issued Date: Mon, 31 Oct 2011 14:27:07 -0500 (31/10/11 19:27:07) I have been struggling with this same problem for the past 2 weeks and have been unable to resolve it as of yet. I would appreciate anyone's help!! We actually had a user report this with the DSpace feedback feature on our site. Thanks a bunch, Sue Sue Walker-Thornton (757) 864-2368 -Original Message- From: James Bardin [mailto:jbar...@bu.edu] Sent: Thursday, October 27, 2011 1:12 PM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] search can't sort by date issued Hello, Attempting to sort search results by issue date results in the following error: ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued Checking DSQuery.java (line ~235) shows that this is catching a generic exception from lucene, which results in the sorting falling back to relevance. This is confusing to users, because there no indication from the xmlui that would indicate the sorting failed or is incorrect. Any ideas why sorting on dc.date.isssued is failing, or where to start for troubleshooting? Thanks, -- James Bardin jbar...@bu.edu Systems Engineer Boston University IST -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech ** ** -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
On Tue, Dec 20, 2011 at 8:46 AM, Bram Luyten b...@mire.be wrote: My very wild guess would be that it's related to a value in date issued that breaks the creation of the index on this value. That's a very good guess! I haven't yet gotten around to building a dspace with some more debugging around the search code to see what's actually happening, but this is very plausible. Looking the our DB, there were a few non-ISO format dates, and a couple nulls. I updates the bad format dates, removed the nulls, and reindexed, but search by date issued still doesn't work. Do all the dates need to be in a more specific format? (like full ISO: 2000-01-12T12:13:14Z). Thanks, -- James Bardin jbar...@bu.edu Systems Engineer Boston University IST -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
I believe I saw an issue like this a while back in 1.6.2, when issue date was missing the trailing ‘Z’. -- Joe DeVries Senior Software Engineer Digital Library Services, TDL University of Texas at Austin 512-495-4639 PCL 1.335 / S5477 From: James Bardin [mailto:jbar...@bu.edu] Sent: Tuesday, December 20, 2011 9:38 AM To: Bram Luyten Cc: dspace-tech@lists.sourceforge.net; susan.m.thorn...@nasa.gov Subject: Re: [Dspace-tech] search can't sort by date issued On Tue, Dec 20, 2011 at 8:46 AM, Bram Luyten b...@mire.bemailto:b...@mire.be wrote: My very wild guess would be that it's related to a value in date issued that breaks the creation of the index on this value. That's a very good guess! I haven't yet gotten around to building a dspace with some more debugging around the search code to see what's actually happening, but this is very plausible. Looking the our DB, there were a few non-ISO format dates, and a couple nulls. I updates the bad format dates, removed the nulls, and reindexed, but search by date issued still doesn't work. Do all the dates need to be in a more specific format? (like full ISO: 2000-01-12T12:13:14Z). Thanks, -- James Bardin jbar...@bu.edumailto:jbar...@bu.edu Systems Engineer Boston University IST -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
On Tue, Dec 20, 2011 at 10:41 AM, DeVries, Joe joe.devr...@austin.utexas.edu wrote: I believe I saw an issue like this a while back in 1.6.2, when issue date was missing the trailing ‘Z’. That would be unfortunate, since a date doesn't *require* the time portion (or the Zulu). Many of the older items in our db don't even have a specific day, just or -MM (both of which should be compliant). -- James Bardin jbar...@bu.edu Systems Engineer Boston University IST -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
Interesting. I just looked at Items I just imported and I did NOT specify what any of these 3 fields should be in dublin_core.xml – they all got set by DSpace (version 1.7.1). It’s interesting that date.issued is the only one with a different format: DC Field Value Language dc.date.accessioned2011-12-20T18:59:53Z - dc.date.available 2011-12-20T18:59:53Z - dc.date.issued 2011-12-20 - I wonder if maybe this is the problem….? Sue Sue Walker-Thornton (757) 864-2368 From: DeVries, Joe [mailto:joe.devr...@austin.utexas.edu] Sent: Tuesday, December 20, 2011 10:42 AM To: James Bardin; Bram Luyten Cc: dspace-tech@lists.sourceforge.net; Thornton, Susan M. (LARC-B702)[LITES] Subject: RE: [Dspace-tech] search can't sort by date issued I believe I saw an issue like this a while back in 1.6.2, when issue date was missing the trailing ‘Z’. -- Joe DeVries Senior Software Engineer Digital Library Services, TDL University of Texas at Austin 512-495-4639 PCL 1.335 / S5477 From: James Bardin [mailto:jbar...@bu.edu] Sent: Tuesday, December 20, 2011 9:38 AM To: Bram Luyten Cc: dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net; susan.m.thorn...@nasa.govmailto:susan.m.thorn...@nasa.gov Subject: Re: [Dspace-tech] search can't sort by date issued On Tue, Dec 20, 2011 at 8:46 AM, Bram Luyten b...@mire.bemailto:b...@mire.be wrote: My very wild guess would be that it's related to a value in date issued that breaks the creation of the index on this value. That's a very good guess! I haven't yet gotten around to building a dspace with some more debugging around the search code to see what's actually happening, but this is very plausible. Looking the our DB, there were a few non-ISO format dates, and a couple nulls. I updates the bad format dates, removed the nulls, and reindexed, but search by date issued still doesn't work. Do all the dates need to be in a more specific format? (like full ISO: 2000-01-12T12:13:14Z). Thanks, -- James Bardin jbar...@bu.edumailto:jbar...@bu.edu Systems Engineer Boston University IST -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
On Tue, Dec 20, 2011 at 20:55, Thornton, Susan M. (LARC-B702)[LITES] susan.m.thorn...@nasa.gov wrote: dc.date.accessioned 2011-12-20T18:59:53Z - dc.date.available 2011-12-20T18:59:53Z - dc.date.issued2011-12-20 - I wonder if maybe this is the problem….? Sue, this is normal behavior. Regards, ~~helix84 -- Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] search can't sort by date issued
Hello, Attempting to sort search results by issue date results in the following error: ERROR org.dspace.search.DSQuery @ Unable to use speficied sort option: dateissued Checking DSQuery.java (line ~235) shows that this is catching a generic exception from lucene, which results in the sorting falling back to relevance. This is confusing to users, because there no indication from the xmlui that would indicate the sorting failed or is incorrect. Any ideas why sorting on dc.date.isssued is failing, or where to start for troubleshooting? Thanks, -- James Bardin jbar...@bu.edu Systems Engineer Boston University IST -- The demand for IT networking professionals continues to grow, and the demand for specialized networking skills is growing even more rapidly. Take a complimentary Learning@Cisco Self-Assessment and learn about Cisco certifications, training, and career opportunities. http://p.sf.net/sfu/cisco-dev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] search can't sort by date issued
On Thu, Oct 27, 2011 at 1:52 PM, Blanco, Jose blan...@umich.edu wrote: # Browse indexes webui.browse.index.1 = title:item:title webui.browse.index.2 = author:metadata:dc.contributor.author:text webui.browse.index.3 = subject:metadata:dc.subject.*:text webui.browse.index.4 = dateissued:item:dateissued # Sorting options webui.itemlist.sort-option.1 = title:dc.title:title webui.itemlist.sort-option.2 = dateissued:dc.date.issued:date webui.itemlist.sort-option.3 = dateaccessioned:dc.date.accessioned:date Yeah, I have dateissued in both the browse.index and sort-option, like above. Sorting by dateissued *does* work in browsing, but not for search results (I think search result ordering is done by lucene, and not the webui). I took a guess and added another search index for dateissued:dc.date.issued:date, but that doesn't seem to have any effect. -jim -- The demand for IT networking professionals continues to grow, and the demand for specialized networking skills is growing even more rapidly. Take a complimentary Learning@Cisco Self-Assessment and learn about Cisco certifications, training, and career opportunities. http://p.sf.net/sfu/cisco-dev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
