Re: [Dspace-tech] Administrative Statistics
Hi, Continuing the tangent: We found it confusing when we compared item views for the top items in our repository: the numbers on the site-wide stats don't match the item-level stats. Immediately this reduced our confidence in both stats pages: what exactly is being counted and are these reliable sources to report on? After some digging, we were surprised to learn that: 1) DSpace generates and displays stats from two different sources, 2) what's termed legacy stats is actually the current source for site-level stats, 3) traffic from bots inflate stats in both Solr and log-based stats. These are not things that are obvious in the UI or the documentation. At our institution, the repository manager is a non-technical person that (quite reasonably) took the stats at face value. She did not expect to need a long explanation from the system administrator (myself) on how that stats actually work. Moving forward, it would be preferable if the same source were used for all stats displayed in the UI (site-wide and item-level). Further, the site-wide stats could be reviewed and brought up to date (log processing time? no thanks). Cheers, Anthony -Original Message- From: Mark H. Wood [mailto:mw...@iupui.edu] Sent: Wednesday, August 19, 2015 8:55 AM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Administrative Statistics On Tue, Aug 18, 2015 at 02:12:18PM -0500, Tim Donohue wrote: The Administrative Statistics are the (very old) legacy DSpace statistics pulled from log files, which pre-dated the Usage Statistics (based on Solr). They are only generated by running these commandline options: [dspace]/bin/dspace stat-initial [dspace]/bin/dspace stat-general [dspace]/bin/dspace stat-monthly The only reason they still exist is that the Usage Statistics don't provide all the same information (yet). The Usage Statistics are much more accurate in providing usage information (as these legacy, log-based stats do not filter out spiders or similar). But, the legacy, log-based stats do provide some unique administrative statistics, like the counts of the number of actions performed in your DSpace, etc. [tangent] I don't find the situation confusing at all. Service administrators have different needs than contributors and editors. While the mechanism for gathering and storing sitewide admin. statistics might be improvable, I think we ought to look at bringing them up to date and fleshing them out with other information that admin.s would want. As an example, people interested in the content will appreciate having robot accesses filtered out, but admin.s might profit from seeing filtered and unfiltered counts side by side. Even more so if they can sample these counts mechanically and accumulate them for visualization. Some other stuff I get asked for includes simple counts of how much stuff we have: how many Items, how many Bitstreams, how many image Bitstreams. End users don't care about such things, but senior administrators do. For the future, another thing that our biggest statistical consumers want very much is views aggregated by *author*. I'm looking forward to first-class support for author identities so that we can do this well. [tired old refrain] Statistics doesn't mean the same thing to everyone. It may not mean the same thing to *anyone*. -- Mark H. Wood Lead Technology Analyst University Library Indiana University - Purdue University Indianapolis 755 W. Michigan Street Indianapolis, IN 46202 317-274-0749 www.ulib.iupui.edu -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Report of items including usage statistics
Hello, We're interested in creating a report (spreadsheet) of items, which includes basic metadata AND associated usage statistics. For instance: - Title - Author - Abstract - Date Created - Owning Collection - Total Views (since accessioned) - Total Downloads (all bitstreams) What's the best way to do this? Best, Anthony -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Report of items including usage statistics
Hi Terry, PHP works for me. :) Thanks for your overview of the 2 approaches. The first one looks easier so I’ll try that one. Best, Anthony From: Terry Brady [mailto:terry.br...@georgetown.edu] Sent: Tuesday, August 04, 2015 12:52 PM To: Anthony Petryk anthony.pet...@uottawa.ca Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Report of items including usage statistics Do you have a preferred technology stack for the solution? I have some PHP code that may be useful. The total views, total downloads, and owning collection id can be pulled from the solr statistics repository. * Query solr for item views, facet by item id * Query solr for bitstream downloads, facet by item id (do you want to include thumbnail views?) The title, author, abstract, and date created are probably easiest to pull from the database. Here are 2 approaches that would work. 1. Query the database for all items. As you iterate over the SQL results, query SOLR for the view/download counts 2. Run the faceted SOLR queries by item number. As you iterate over the XML/JSON results, query the database for supplemental metadata. Terry On Tue, Aug 4, 2015 at 8:29 AM, Anthony Petryk anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca wrote: Hello, We’re interested in creating a report (spreadsheet) of items, which includes basic metadata AND associated usage statistics. For instance: - Title - Author - Abstract - Date Created - Owning Collection - Total Views (since accessioned) - Total Downloads (all bitstreams) What’s the best way to do this? Best, Anthony -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Terry Brady Applications Programmer Analyst Georgetown University Library Information Technology https://www.library.georgetown.edu/lit/code 425-298-5498 (Seattle, WA) -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Report of items including usage statistics
Thanks very much, Terry. Anthony From: Terry Brady [mailto:terry.br...@georgetown.edu] Sent: Tuesday, August 04, 2015 2:09 PM To: Anthony Petryk anthony.pet...@uottawa.ca Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Report of items including usage statistics Anthony, Check out the following code. There might be some useful code to clone. * Statistics report tool (this code reports collection by collection): https://github.com/Georgetown-University-Libraries/batch-tools/blob/master/web/stats/qcHierarchyStats.php * Populate a PHP array with the collection list using SQL: https://github.com/Georgetown-University-Libraries/batch-tools/blob/master/web/community.php * Package a SOLR query with an ajax call: https://github.com/Georgetown-University-Libraries/batch-tools/blob/master/web/stats/qcHierarchyStats.php#L41-L50 Here is a wiki page describing our statistics reports. * https://github.com/Georgetown-University-Libraries/batch-tools/wiki/Statistics-reporting Good luck, Terry On Tue, Aug 4, 2015 at 10:45 AM, Anthony Petryk anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca wrote: Hi Terry, PHP works for me. :) Thanks for your overview of the 2 approaches. The first one looks easier so I’ll try that one. Best, Anthony From: Terry Brady [mailto:terry.br...@georgetown.edumailto:terry.br...@georgetown.edu] Sent: Tuesday, August 04, 2015 12:52 PM To: Anthony Petryk anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca Cc: dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Report of items including usage statistics Do you have a preferred technology stack for the solution? I have some PHP code that may be useful. The total views, total downloads, and owning collection id can be pulled from the solr statistics repository. * Query solr for item views, facet by item id * Query solr for bitstream downloads, facet by item id (do you want to include thumbnail views?) The title, author, abstract, and date created are probably easiest to pull from the database. Here are 2 approaches that would work. 1. Query the database for all items. As you iterate over the SQL results, query SOLR for the view/download counts 2. Run the faceted SOLR queries by item number. As you iterate over the XML/JSON results, query the database for supplemental metadata. Terry On Tue, Aug 4, 2015 at 8:29 AM, Anthony Petryk anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca wrote: Hello, We’re interested in creating a report (spreadsheet) of items, which includes basic metadata AND associated usage statistics. For instance: - Title - Author - Abstract - Date Created - Owning Collection - Total Views (since accessioned) - Total Downloads (all bitstreams) What’s the best way to do this? Best, Anthony -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Terry Brady Applications Programmer Analyst Georgetown University Library Information Technology https://www.library.georgetown.edu/lit/code 425-298-5498tel:425-298-5498 (Seattle, WA) -- Terry Brady Applications Programmer Analyst Georgetown University Library Information Technology https://www.library.georgetown.edu/lit/code 425-298-5498 (Seattle, WA) -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] stats-util error Document is missing mandatory uniqueKey field: uid
Many thanks for your help.
Anthony
From: Andrea Schweer [mailto:schw...@waikato.ac.nz]
Sent: Thursday, July 23, 2015 5:44 PM
To: Anthony Petryk anthony.pet...@uottawa.ca;
dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] stats-util error Document is missing mandatory
uniqueKey field: uid
Hi Anthony,
oh right sorry, I should have looked at the help for -r: While indexing the
bundle names remove the statistics about deleted bitstreams. So yes, it looks
like an upgrade is your best bet.
Alternatively, you could write a curation task / script that goes through all
bitstream IDs in your statistics solr core and for each one, checks whether
it's still a valid bitstream. If not, delete all hits for this bitstream by
sending the appropriate solr query. Sorry, I don't have time right now to dig
up examples for these queries, but if you're interested in going down that
route and need help with the queries, post to the mailing list again and I'm
sure someone will respond.
cheers,
Andrea
On 24/07/15 03:41, Anthony Petryk wrote:
Hi again Andrea,
As you predicted, I now get a version conflict error after applying your uid
fix and rebuilding:
Exception: version conflict for expected=1507325787224670208
actual=1507501635509682176
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version
conflict for expected=1507325787224670208 actual=1507501635509682176
So it looks like we'll need to upgrade to 5 for the full solution.
Cheers,
Anthony
From: Anthony Petryk [mailto:anthony.pet...@uottawa.ca]
Sent: Thursday, July 23, 2015 11:22 AM
To: Andrea Schweer schw...@waikato.ac.nzmailto:schw...@waikato.ac.nz;
dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] stats-util error Document is missing mandatory
uniqueKey field: uid
Thanks for the response, Andrea.
Do I need to do anything after updating the solrconfig.xml file, as you
suggested?
I tried running but stats-util with just the -r flag, but it doesn't work. I
confirmed this by looking at StatisticsClient.java, which only uses r if b is
present:
else if(line.hasOption('b'))
{
SolrLogger.reindexBitstreamHits(line.hasOption('r'));
}
Best,
Anthony
From: Andrea Schweer [mailto:schw...@waikato.ac.nz]
Sent: Wednesday, July 22, 2015 5:32 PM
To: Anthony Petryk
anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca;
dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] stats-util error Document is missing mandatory
uniqueKey field: uid
Hi Anthony,
On 23/07/15 01:37, Anthony Petryk wrote:
DSpace 4.3. We're getting an error when trying to run dspace stats-util -br:
Document is missing mandatory uniqueKey field: uid. We want to run this
command specifically to remove stats for deleted bitstreams (-r option).
This sounds like the same root problem as
DS-2212https://jira.duraspace.org/browse/DS-2212, which is fixed in version
5.2. But I just wanted to check: is there any way to get this to work without
upgrading?
You're actually running into https://jira.duraspace.org/browse/DS-2489. If I
remember correctly then you may be able to fix this by applying this small
change to the solrconfig.xml for the statistics core:
https://github.com/DSpace/DSpace/commit/b07707874a489b4d5238c87d04d2be1b28ece447
However, chances are that you'll then just run into further issues such as
DS-2212.
If you're only after removing deleted bitstreams, try leaving out the b flag
(Reindex the bitstreams to ensure we have the bundle name). Re-indexing will
hit the uid bug (and probably the _version_ one too); deleting (-r only)
shouldn't have the same issue since it won't try adding documents to the solr
index.
cheers,
Andrea
--
Dr Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
--
Dr Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
+64-7-837 9120
--
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] stats-util error Document is missing mandatory uniqueKey field: uid
Thanks for the response, Andrea.
Do I need to do anything after updating the solrconfig.xml file, as you
suggested?
I tried running but stats-util with just the -r flag, but it doesn't work. I
confirmed this by looking at StatisticsClient.java, which only uses r if b is
present:
else if(line.hasOption('b'))
{
SolrLogger.reindexBitstreamHits(line.hasOption('r'));
}
Best,
Anthony
From: Andrea Schweer [mailto:schw...@waikato.ac.nz]
Sent: Wednesday, July 22, 2015 5:32 PM
To: Anthony Petryk anthony.pet...@uottawa.ca;
dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] stats-util error Document is missing mandatory
uniqueKey field: uid
Hi Anthony,
On 23/07/15 01:37, Anthony Petryk wrote:
DSpace 4.3. We're getting an error when trying to run dspace stats-util -br:
Document is missing mandatory uniqueKey field: uid. We want to run this
command specifically to remove stats for deleted bitstreams (-r option).
This sounds like the same root problem as
DS-2212https://jira.duraspace.org/browse/DS-2212, which is fixed in version
5.2. But I just wanted to check: is there any way to get this to work without
upgrading?
You're actually running into https://jira.duraspace.org/browse/DS-2489. If I
remember correctly then you may be able to fix this by applying this small
change to the solrconfig.xml for the statistics core:
https://github.com/DSpace/DSpace/commit/b07707874a489b4d5238c87d04d2be1b28ece447
However, chances are that you'll then just run into further issues such as
DS-2212.
If you're only after removing deleted bitstreams, try leaving out the b flag
(Reindex the bitstreams to ensure we have the bundle name). Re-indexing will
hit the uid bug (and probably the _version_ one too); deleting (-r only)
shouldn't have the same issue since it won't try adding documents to the solr
index.
cheers,
Andrea
--
Dr Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
--
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] stats-util error Document is missing mandatory uniqueKey field: uid
Hi again Andrea,
As you predicted, I now get a version conflict error after applying your uid
fix and rebuilding:
Exception: version conflict for expected=1507325787224670208
actual=1507501635509682176
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version
conflict for expected=1507325787224670208 actual=1507501635509682176
So it looks like we'll need to upgrade to 5 for the full solution.
Cheers,
Anthony
From: Anthony Petryk [mailto:anthony.pet...@uottawa.ca]
Sent: Thursday, July 23, 2015 11:22 AM
To: Andrea Schweer schw...@waikato.ac.nz; dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] stats-util error Document is missing mandatory
uniqueKey field: uid
Thanks for the response, Andrea.
Do I need to do anything after updating the solrconfig.xml file, as you
suggested?
I tried running but stats-util with just the -r flag, but it doesn't work. I
confirmed this by looking at StatisticsClient.java, which only uses r if b is
present:
else if(line.hasOption('b'))
{
SolrLogger.reindexBitstreamHits(line.hasOption('r'));
}
Best,
Anthony
From: Andrea Schweer [mailto:schw...@waikato.ac.nz]
Sent: Wednesday, July 22, 2015 5:32 PM
To: Anthony Petryk
anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca;
dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] stats-util error Document is missing mandatory
uniqueKey field: uid
Hi Anthony,
On 23/07/15 01:37, Anthony Petryk wrote:
DSpace 4.3. We're getting an error when trying to run dspace stats-util -br:
Document is missing mandatory uniqueKey field: uid. We want to run this
command specifically to remove stats for deleted bitstreams (-r option).
This sounds like the same root problem as
DS-2212https://jira.duraspace.org/browse/DS-2212, which is fixed in version
5.2. But I just wanted to check: is there any way to get this to work without
upgrading?
You're actually running into https://jira.duraspace.org/browse/DS-2489. If I
remember correctly then you may be able to fix this by applying this small
change to the solrconfig.xml for the statistics core:
https://github.com/DSpace/DSpace/commit/b07707874a489b4d5238c87d04d2be1b28ece447
However, chances are that you'll then just run into further issues such as
DS-2212.
If you're only after removing deleted bitstreams, try leaving out the b flag
(Reindex the bitstreams to ensure we have the bundle name). Re-indexing will
hit the uid bug (and probably the _version_ one too); deleting (-r only)
shouldn't have the same issue since it won't try adding documents to the solr
index.
cheers,
Andrea
--
Dr Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
--
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] stats-util error Document is missing mandatory uniqueKey field: uid
Hello, DSpace 4.3. We're getting an error when trying to run dspace stats-util -br: Document is missing mandatory uniqueKey field: uid. We want to run this command specifically to remove stats for deleted bitstreams (-r option). This sounds like the same root problem as DS-2212https://jira.duraspace.org/browse/DS-2212, which is fixed in version 5.2. But I just wanted to check: is there any way to get this to work without upgrading? Thanks, Anthony -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Multiple File Download stats for single file
Hello, We've noticed that the item statistics page sometimes lists the same file multiple times in the File Downloads section, each with a different number of downloads. For example: File1.pdf 45 File2.mp423 File1.pdf 3 Querying Solr directly, it looks like there's multiple bitstream IDs for the same file, which explains why they're being reported as different files. I suspect that this happens when a submitter re-uploads a file with the same name during submission workflow (and maybe after it's been installed). This is not unusual in our case because we have strict guidelines for things like the title page, which we require the submitter to fix before the submission is approved. A similar problem occurs when the submitter decides to change the filename slightly, even though it's conceptually the same file: File1.pdf 45 File1_Final.pdf 167 In both cases, we only want to show File Download statistics for the files that actually appear on the Item Page. Is there an easy way to do this, or should we submit Jira issue? Thanks, Anthony -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] How to translate text in Dspace 4.2 JSPUI
Hi, Check your DSpace source directory for the following file: [dspace-src]/dspace/modules/jspui/src/main/resources/Messages_es.properties. If it's not there, you can download it from here: https://github.com/DSpace/dspace-api-lang/blob/master/src/main/resources/Messages_es.properties. Then, you'll need to add the following entries to the file and translate them: jsp.search.facet.refine = Discover jsp.search.facet.refine.author = Authors jsp.search.facet.refine.subject = Subjects jsp.search.facet.refine.dateIssued = Date Issued Finally, rebuild DSpace and it should pick up your translations. As for the See button, the text appears to be hard-coded into the following file (line 149): [dspace-src]/dspace-jspui/src/main/webapp/home.jsp. You can either translate it there directly, or (better) replace 'See' with a new token fmt:message key=jsp.home.see/, add this a new entry for jsp.home.see to your Messages_es.properties file. Then rebuild. I hope this helps. Anthony -Original Message- From: cbeltran [mailto:cbelt...@humboldt.org.co] Sent: Thursday, May 21, 2015 5:37 PM To: dspace-tech@lists.sourceforge.net Subject: [Dspace-tech] How to translate text in Dspace 4.2 JSPUI Hi Everybody, I need translate texts to spanish language in my dspace installation version 4.2, interface JSPUI over centos server. how I do? Attached picture with text i need translate. http://dspace.2283337.n4.nabble.com/file/n4677976/dspacetranslate.png Thanks. Regards. -- View this message in context: http://dspace.2283337.n4.nabble.com/How-to-translate-text-in-Dspace-4-2-JSPUI-tp4677976.html Sent from the DSpace - Tech mailing list archive at Nabble.com. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] How to translate text in Dspace 4.2 JSPUI
Hi again, See section 2.4.2 Overview of DSpace Directories of the DSpace 4.x documentation for details. When you download DSpace 4.2, it doesn't have a resources directory under [dspace-src]/dspace/modules/jspui/src/main/, so you can just create it. Then you can put all your Messages.properties files in there. Anthony -Original Message- From: cbeltran [mailto:cbelt...@humboldt.org.co] Sent: Friday, May 22, 2015 11:32 AM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] How to translate text in Dspace 4.2 JSPUI Hi Anthony Thanks for your reply. On my dspace installation not exists this path: [dspace-src]/dspace/modules/jspui/src/main/resources/ How do I? Thanks -- View this message in context: http://dspace.2283337.n4.nabble.com/How-to-translate-text-in-Dspace-4-2-JSPUI-tp4677976p4677993.html Sent from the DSpace - Tech mailing list archive at Nabble.com. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] How to translate text in Dspace 4.2 JSPUI
You have to rebuild DSpace for the changes to be picked up; just restarting Tomcat isn't enough. There are instructions for building DSpace in the docs. Once you've rebuilt DSpace, you would then deploy it to your /web deploy directory. Anthony -Original Message- From: cbeltran [mailto:cbelt...@humboldt.org.co] Sent: Friday, May 22, 2015 12:35 PM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] How to translate text in Dspace 4.2 JSPUI Hi Anthony Thanks. I dont understand, I have two directories: /dspace-4.2-src-release -- Source code and /web -- Deploy Now, in this path: /home/dspace/dspace-4.2-src-release/dspace-api/src/main/resources, found Messages.properties After create directory resources in path: /home/dspace/dspace-4.2-src-release/dspace/modules/jspui/src/main, and copy Messages.properties and Messages_es.properties Restart tomcat, but not work the translation to spanish. I'm doing the right process? In the web / directory I have to do something extra? Im sorry. Regards -- View this message in context: http://dspace.2283337.n4.nabble.com/How-to-translate-text-in-Dspace-4-2-JSPUI-tp4677976p4677997.html Sent from the DSpace - Tech mailing list archive at Nabble.com. -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] spider ip recognition
Thanks for the info Hardy. I just discovered that Jira issue yesterday. I'll
probably use your approach for our own stats, but I'm sure other sites would
benefit from domain/agent handling when running stats-util -m or stats-util
-i (as described in the issue).
Best,
Anthony
From: Pottinger, Hardy J. [mailto:pottinge...@missouri.edu]
Sent: Friday, May 15, 2015 10:19 AM
To: Anthony Petryk; Monika C. Mevenkamp; dspace-tech@lists.sourceforge.net
Subject: RE: [Dspace-tech] spider ip recognition
Hi, you've run into a known issue, and one I very recently wrestled with myself:
https://jira.duraspace.org/browse/DS-2431
See my last comment on that ticket, I found a way around the issue, by simply
deleting the spider docs from the stats index via a query in the Solr admin
interface.
--Hardy
From: Anthony Petryk [anthony.pet...@uottawa.ca]
Sent: Thursday, May 14, 2015 12:06 PM
To: Monika C. Mevenkamp;
dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] spider ip recognition
Hi again,
Unfortunately, the documentation for the stats-util command is incorrect.
Specifically this line:
-i or --delete-spiders-by-ip: Delete Spiders in Solr By IP Address, DNS name,
or Agent name. Will prune out all records that match spider identification
patterns.
Running stats-util -i does not actually remove spiders by DNS name or Agent
name. Here's are the relevant sections of the code, from StatisticsClient.java
and SolrLogger.java:
(...)
else if(line.hasOption('i'))
{
SolrLogger.deleteRobotsByIP();
}
public static void deleteRobotsByIP()
{
for(String ip : SpiderDetector.getSpiderIpAddresses()){
deleteIP(ip);
}
}
What this means is that, if a spider is in your Solr stats, there's no way to
remove it other than manually adding its IP to [dpsace]/config/spiders; adding
its DNS name or Agent name to the configs will not expunge it. Updating the
spider files with stats-util -u does little to help because the IP lists it
pulls from are out of date.
An example is the spider from the Bing search engine: bingbot. As of DSpace
4.3, it's not in the list of spiders by DNS name or Agent name, nor is it in
the list of spider IP addresses. So anyone running DSpace 4.3 likely has usage
stats inflated by visits from this spider. The only way to remove it is to
specify all the IPs for bingbot. Multiply that by all the other new spiders
and we're talking about a lot of work.
I tried briefly to modify the code to take domains/agents into account when
marking or deleting spiders, but I wasn't able to figure out how to query Solr
with regex patterns. It's easier to do with IPs because each IP or IP range is
transformed into a String and used as a standard query parameter.
Anthony
From: Monika C. Mevenkamp [mailto:moni...@princeton.edu]
Sent: Thursday, May 14, 2015 11:17 AM
To: Anthony Petryk
Cc: Monika C. Mevenkamp;
dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] spider ip recognition
Anthony
Since dspace 4 you can filter by userAgent
see
https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics+Maintenance#SOLRStatisticsMaintenance-FilteringandPruningSpiders
I have not used this myself and am not sure whether these filters are applied
as crawlers access content - or whether you need to run the
[dspace]/bin/dspace stats-util command on a regular basis. You definitely need
to run it to prune mark usage events after you configure
a list of userAgents you want to filter against.
Monika
Monika Mevenkamp
phone: 609-258-4161
Princeton University, Princeton, NJ 08544
On May 12, 2015, at 2:13 PM, Anthony Petryk
anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca wrote:
After a bit of investigation, it turns out that a significant portion of our
items stats come from spiders. Any thoughts on the best way to go about
removing them from Solr retroactively? There's nothing that I can see in the
code that will do this by domain or agent, only IP. We're not excited at the
prospect of pulling out the IPs of all the spiders in order run stats-util -i
effectively.
Cheers,
Anthony
From: Monika C. Mevenkamp [mailto:moni...@princeton.edu]
Sent: Friday, May 08, 2015 9:59 AM
To: Anthony Petryk
Cc: dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] spider ip recognition
Anthony
I wrote a small ruby script to put solr queries together when I was poking
around my stats
see https://github.com/akinom/dscriptor/blob/master/solr/solr_query.rb
an example parameter file is
https://github.com/akinom/dscriptor/blob/master/solr/solr_query.yml
run it asruby solr/solr_query.rb
of cause you ned to adjust the parameters in the mL file
you can query like this
http://localhost:YOUR-PORT/solr/statistics/select?wt=jsonindent=truerows=1facet
Re: [Dspace-tech] spider ip recognition
Hi again,
Unfortunately, the documentation for the stats-util command is incorrect.
Specifically this line:
-i or --delete-spiders-by-ip: Delete Spiders in Solr By IP Address, DNS name,
or Agent name. Will prune out all records that match spider identification
patterns.
Running “stats-util –i” does not actually remove spiders by DNS name or Agent
name. Here’s are the relevant sections of the code, from StatisticsClient.java
and SolrLogger.java:
(…)
else if(line.hasOption('i'))
{
SolrLogger.deleteRobotsByIP();
}
public static void deleteRobotsByIP()
{
for(String ip : SpiderDetector.getSpiderIpAddresses()){
deleteIP(ip);
}
}
What this means is that, if a spider is in your Solr stats, there’s no way to
remove it other than manually adding its IP to [dpsace]/config/spiders; adding
its DNS name or Agent name to the configs will not expunge it. Updating the
spider files with “stats-util –u” does little to help because the IP lists it
pulls from are out of date.
An example is the spider from the Bing search engine: bingbot. As of DSpace
4.3, it’s not in the list of spiders by DNS name or Agent name, nor is it in
the list of spider IP addresses. So anyone running DSpace 4.3 likely has usage
stats inflated by visits from this spider. The only way to remove it is to
specify all the IPs for bingbot. Multiply that by all the other “new” spiders
and we’re talking about a lot of work.
I tried briefly to modify the code to take domains/agents into account when
marking or deleting spiders, but I wasn’t able to figure out how to query Solr
with regex patterns. It’s easier to do with IPs because each IP or IP range is
transformed into a String and used as a standard query parameter.
Anthony
From: Monika C. Mevenkamp [mailto:moni...@princeton.edu]
Sent: Thursday, May 14, 2015 11:17 AM
To: Anthony Petryk
Cc: Monika C. Mevenkamp; dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] spider ip recognition
Anthony
Since dspace 4 you can filter by userAgent
see
https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics+Maintenance#SOLRStatisticsMaintenance-FilteringandPruningSpiders
I have not used this myself and am not sure whether these filters are applied
as crawlers access content - or whether you need to run the
[dspace]/bin/dspace stats-util command on a regular basis. You definitely need
to run it to prune mark usage events after you configure
a list of userAgents you want to filter against.
Monika
Monika Mevenkamp
phone: 609-258-4161
Princeton University, Princeton, NJ 08544
On May 12, 2015, at 2:13 PM, Anthony Petryk
anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca wrote:
After a bit of investigation, it turns out that a significant portion of our
items stats come from spiders. Any thoughts on the best way to go about
removing them from Solr retroactively? There’s nothing that I can see in the
code that will do this by domain or agent, only IP. We’re not excited at the
prospect of pulling out the IPs of all the spiders in order run “stats-util –i”
effectively.
Cheers,
Anthony
From: Monika C. Mevenkamp [mailto:moni...@princeton.edu]
Sent: Friday, May 08, 2015 9:59 AM
To: Anthony Petryk
Cc: dspace-tech@lists.sourceforge.netmailto:dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] spider ip recognition
Anthony
I wrote a small ruby script to put solr queries together when I was poking
around my stats
see https://github.com/akinom/dscriptor/blob/master/solr/solr_query.rb
an example parameter file is
https://github.com/akinom/dscriptor/blob/master/solr/solr_query.yml
run it asruby solr/solr_query.rb
of cause you ned to adjust the parameters in the mL file
you can query like this
http://localhost:YOUR-PORT/solr/statistics/select?wt=jsonindent=truerows=1facet=truefacet.field=ipfacet.mincount=1q=type:2+id:218+isBot:false
exclude records that are marked as bots
do type:2 - aka items
do id:218 - aka item with id 218
return one item
facet on ip addresses
crank up the number of rows to get more matching docs
Monika
Monika Mevenkamp
phone: 609-258-4161
Princeton University, Princeton, NJ 08544
On May 7, 2015, at 3:26 PM, Anthony Petryk
anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca wrote:
Anyway, we want to determine whether these stats are bona fide or whether
there's something wrong with the spider detection. From the documentation it
seems we have to query Solr directly to do this. Not being an expert in Solr,
I'm hoping someone on this list could provide the query that retrieves *all the
stats for a given item* (i.e. what's listed under Common stored fields for all
usage events in the documentation).
--
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Re: [Dspace-tech] spider ip recognition
After a bit of investigation, it turns out that a significant portion of our items stats come from spiders. Any thoughts on the best way to go about removing them from Solr retroactively? There's nothing that I can see in the code that will do this by domain or agent, only IP. We're not excited at the prospect of pulling out the IPs of all the spiders in order run stats-util -i effectively. Cheers, Anthony From: Monika C. Mevenkamp [mailto:moni...@princeton.edu] Sent: Friday, May 08, 2015 9:59 AM To: Anthony Petryk Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] spider ip recognition Anthony I wrote a small ruby script to put solr queries together when I was poking around my stats see https://github.com/akinom/dscriptor/blob/master/solr/solr_query.rb an example parameter file is https://github.com/akinom/dscriptor/blob/master/solr/solr_query.yml run it asruby solr/solr_query.rb of cause you ned to adjust the parameters in the mL file you can query like this http://localhost:YOUR-PORT/solr/statistics/select?wt=jsonindent=truerows=1facet=truefacet.field=ipfacet.mincount=1q=type:2+id:218+isBot:false exclude records that are marked as bots do type:2 - aka items do id:218 - aka item with id 218 return one item facet on ip addresses crank up the number of rows to get more matching docs Monika Monika Mevenkamp phone: 609-258-4161 Princeton University, Princeton, NJ 08544 On May 7, 2015, at 3:26 PM, Anthony Petryk anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca wrote: Anyway, we want to determine whether these stats are bona fide or whether there's something wrong with the spider detection. From the documentation it seems we have to query Solr directly to do this. Not being an expert in Solr, I'm hoping someone on this list could provide the query that retrieves *all the stats for a given item* (i.e. what's listed under Common stored fields for all usage events in the documentation). -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] spider ip recognition
Thanks very much Monika! I'll try it out. Cheers, Anthony From: Monika C. Mevenkamp [mailto:moni...@princeton.edu] Sent: Friday, May 08, 2015 9:59 AM To: Anthony Petryk Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] spider ip recognition Anthony I wrote a small ruby script to put solr queries together when I was poking around my stats see https://github.com/akinom/dscriptor/blob/master/solr/solr_query.rb an example parameter file is https://github.com/akinom/dscriptor/blob/master/solr/solr_query.yml run it asruby solr/solr_query.rb of cause you ned to adjust the parameters in the mL file you can query like this http://localhost:YOUR-PORT/solr/statistics/select?wt=jsonindent=truerows=1facet=truefacet.field=ipfacet.mincount=1q=type:2+id:218+isBot:false exclude records that are marked as bots do type:2 - aka items do id:218 - aka item with id 218 return one item facet on ip addresses crank up the number of rows to get more matching docs Monika Monika Mevenkamp phone: 609-258-4161 Princeton University, Princeton, NJ 08544 On May 7, 2015, at 3:26 PM, Anthony Petryk anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca wrote: Anyway, we want to determine whether these stats are bona fide or whether there's something wrong with the spider detection. From the documentation it seems we have to query Solr directly to do this. Not being an expert in Solr, I'm hoping someone on this list could provide the query that retrieves *all the stats for a given item* (i.e. what's listed under Common stored fields for all usage events in the documentation). -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] spider ip recognition
We're having a similar problem (DSpace 4.3). Here's an example: http://www.ruor.uottawa.ca/handle/10393/23938/statistics. These stats seem high for this item (although we could be wrong about that), but what's more peculiar is all the traffic from China. We've run through all the options for the stats-util command but the numbers remain the same. Note that when we run stats-util - u (update the IP lists), we always get the message Not modified - so not downloaded. I'm guessing that the spider lists by domain name or user-agent are more current? Anyway, we want to determine whether these stats are bona fide or whether there's something wrong with the spider detection. From the documentation it seems we have to query Solr directly to do this. Not being an expert in Solr, I'm hoping someone on this list could provide the query that retrieves *all the stats for a given item* (i.e. what's listed under Common stored fields for all usage events in the documentation). Thanks for your time, Anthony Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.ca -Original Message- From: Mark H. Wood [mailto:mw...@iupui.edu] Sent: Friday, April 24, 2015 9:42 AM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] spider ip recognition On Thu, Apr 23, 2015 at 05:39:01PM +, Monika C. Mevenkamp wrote: I found a couple of really suspicious numbers in my solr stats, aka lots of entries were marked as isBot=false although the probably should has been isBot=true. In the config file I use spiderips.urls = http://iplists.com/google.txt, \ http://iplists.com/inktomi.txt, \ http://iplists.com/lycos.txt, \ http://iplists.com/infoseek.txt, \ http://iplists.com/altavista.txt, \ http://iplists.com/excite.txt, \ http://iplists.com/northernlight.txt, \ http://iplists.com/misc.txt, \ http://iplists.com/non_engines.txt I could not find downloadable lists for Bing, Baidu, Yahoo. The best I saw was: http://myip.ms/info/bots/Google_Bing_Yahoo_Facebook_etc_Bot_IP_Addresses.html Is that reliable ? Does anybody out there have lists / sources that they can share ? What version of DSpace are you running? Recent versions can also recognize spiders by regular expression matching of the domain name or UserAgent: string. (However, that only works for new entries. I've recently found that some of the tools for loading and grooming the stat.s core don't use SpiderDetector and are oblivious of these newer patterns.) Also: does the dspace code gracefully deal with IP address patterns ? That depends on what is considered graceful. The code (in org.dspace.statistics.util.IPTable) accepts patterns in three forms: 11.22.33.44-11.22.33.55 11.22.33.44 11.22.33 Addresses in the first form may be suffixed with a CIDR mask-length, but it is currently ignored. If I've understood the code, a range (the first form) is assumed to differ only in the fourth octet. It will match all addresses between 44 and 55 within the /24 containing the start of the range. The second form is an exact match of a single address. The third form is a match of the first 24 bits -- an entire Class C subnet. There is no provision for IPv6. -- Mark H. Wood Lead Technology Analyst University Library Indiana University - Purdue University Indianapolis 755 W. Michigan Street Indianapolis, IN 46202 317-274-0749 www.ulib.iupui.edu -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] FW: Embargo not lifting in 4.x after upgrade from 1.8.3
-Original Message- From: Anthony Petryk Sent: Thursday, April 16, 2015 11:24 AM To: 'Hayles, Steven' Subject: RE: [Dspace-tech] Embargo not lifting in 4.x after upgrade from 1.8.3 Hi Steven, We too were unsuccessful in trying to convert the old style embargoes. It didn't matter much because in 4.0 (which we're still using) because the call to lifter.liftEmbargo(context, item) in the liftEmbargo method of EmbargoManager is not commented out. I'm guessing it was removed in 4.2 to prevent ALL custom policies from being wiped out inadvertently. But if the only non-default (non-collection) policies you care about are for embargoes, then it should be safe to reinstate the call lifter.liftEmbargo. Then you can use the embargo-lifter utility as before, at least until all the old-style embargoes have passed. I hope this helps. Anthony Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.ca -Original Message- From: Hayles, Steven [mailto:s...@leicester.ac.uk] Sent: Wednesday, April 15, 2015 7:46 AM To: 'dspace-tech@lists.sourceforge.net' Subject: Re: [Dspace-tech] Embargo not lifting in 4.x after upgrade from 1.8.3 I've just discovered that the way I was testing embargo lifting for content created under 4.2 was not valid. New content embargos are lifted correctly. I was changing the embargo date in metadata, which not sufficient since there's a start date in the anonymous read policy that controls access. As far as I can see, this date is not visible under JSPUI. The remaining question is how to convert old style embargos. Thanks -Original Message- From: Hayles, Steven Sent: 15 April 2015 11:10 To: 'dspace-tech@lists.sourceforge.net' Subject: Embargo not lifting in 4.x after upgrade from 1.8.3 I recently upgraded from 1.8.3 to 4.2 (just before 4.3 was released!). Embargo lifting is modifying metadata correctly, but bitstreams remain inaccessible. Embargoed content created under 1.8.3 has no policies on the original bundle and bitstream(s), so restricting access. Embargoed content created under 4.2 has (in my case) anonymous read policies on the original bundle and bitstream(s). Is there an upgrade step to convert old-style embargoed content to new-style? I tried the following, but to no apparent effect: [DSPACE]/bin/dspace migrate-embargo -a [DSPACE]/bin/dspace embargo-lifter -a -v [DSPACE]/bin/dspace embargo-lifter -c -v I wanted to keep embargo behaviour the same initially, so have item level embargo and run the embargo lifter. It correctly manipulates the metadata fields, but, in 4.2, the call to lifter.liftEmbargo is commented out, so there's no chance of policies being manipulated. For embargoed content from 1.8.3, I can modify the policies manually. What needs to happen to lift the embargo on content from 4.2 where the policies already exist? Is the embargo lifter disrupting the new embargo mechanism? I'm using JSPUI Thanks Steven Hayles Systems Analyst IT Services, University of Leicester, Propsect House, 94 Regent Rd, Leicester, LE1 7DA, UK T: +44 (0)116 229 7950 E: s...@le.ac.uk The Queen's Anniversary Prizes 1994, 2002 2013 THE Awards Winners 2007-2013 Elite without being elitist Follow us on Twitter http://twitter.com/uniofleicester or visit our Facebook page https://facebook.com/UniofLeicester -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Solr errors in logs
Hello, We're seeing a fair number of Solr errors in our DSpace logs, of which we've identified 3 distinct types (examples below). Any explanation or help in resolving these would be appreciated. We're running DSpace 4.0 JSPUI. Anthony -- Type 1 2014-11-01 01:07:35,191 ERROR org.dspace.statistics.SolrLogger @ Server at http://localhost/solr/statistics returned non ok status:500, message:Internal Server Error org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://localhost/solr/statistics returned non ok status:500, message:Internal Server Error at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301) at org.dspace.statistics.SolrLogger$ResultProcessor.execute(SolrLogger.java:713) at org.dspace.statistics.SolrLogger.markRobotsByIP(SolrLogger.java:774) at org.dspace.statistics.util.StatisticsClient.main(StatisticsClient.java:81) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:225) at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:77) Type 2 2014-11-01 00:00:29,054 ERROR org.dspace.discovery.SolrServiceImpl @ Error while writing item to discovery index: 10393/23623 message:Server at http://localhost/solr/search returned non ok status:500, message:Internal Server Error org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://localhost/solr/search returned non ok status:500, message:Internal Server Error at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102) at org.dspace.discovery.SolrServiceImpl.writeDocument(SolrServiceImpl.java:677) at org.dspace.discovery.SolrServiceImpl.buildDocument(SolrServiceImpl.java:1372) at org.dspace.discovery.SolrServiceImpl.indexContent(SolrServiceImpl.java:219) at org.dspace.discovery.SolrServiceImpl.updateIndex(SolrServiceImpl.java:391) at org.dspace.discovery.IndexClient.main(IndexClient.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:225) at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:77) Type 3 2014-12-11 14:34:15,948 ERROR org.dspace.statistics.SolrLoggerUsageEventListener @ (that's it, nothing more) -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Discovery search problems
Hello, DSpace 4.0 JSPUI. 1. A search for climate change (no quotes) produces more results than climate AND change. It looks like the search is using OR instead of the more natural AND. 2. Some search input causes Internal Server Errors, for example mismatched parentheses (opening but no closing) or when there's a colon at the end of the search string. Has anyone succeeded in addressing these problems? Thanks, Anthony -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Login fails when following my dspace link
Hello, JSPUI 4.0. Our users have reported not being able to login to DSpace when following the my dspace link ([dpsace.url]/mydspace), in notification emails for example. The browser is redirected to the login page (/password-login) but the page just refreshes after clicking the submit button. We considered just hard-coding the links in the email templates to /password-login (which work fine for logging in), but other users have reported the same issue when trying to access restricted bitstreams, so it's obviously not a problem specific to mydspace. Could this have something to do with secure pages? We have the following security-constraint in our web.xml for the jspui webapp: url-pattern/password-login/url-pattern Any help is appreciated. Anthony -- ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Bad search input breaks discovery, browse
Hello, DSpace 4.0 JSPUI. When using search or browse, entering certain characters causes an Internal Server Error: - Discovery: Any search with an open parenthesis but no closing parenthesis (or vice versa). For example: 1) indian ocean. - Discovery: Any search with the empty string between parentheses. For example: indian () ocean - Browse (by Title): Any search with two terms in quotes. For example: indian ocean There may be other cases that we haven't come across yet. Best, Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.camailto:apet...@uottawa.ca -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Embargo questions - DSpace 4.0 JSPUI
Hello, As per the subject line: - Is the lift date of an embargoed item displayed anywhere (either publicly or as admin)? - Is there a tool available to modify the lift date, say, to extend an embargo by 3 months? - What's the relationship (if any) between the Private Item flag and the Visible/Embargo (from the AccessStep)? - Do the AccessStep and UploadWithEmbargoStep steps need to be enabled together, or can you combine AccessStep with UploadStep to only allow item-level embargoes? Thanks, Anthony Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.camailto:apet...@uottawa.ca -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Statistics reset after upgrade to 4.0
Hello, After upgrading from 1.8.2 to 4.0 (via 3.2), we've noticed that our item, collection, etc. statistics have been reset: they're now counting up from 0, where we previously had about a year's worth of usage. Is there anything special that needs to be done during the upgrade to retain this data? I couldn't find anything in the 4.0 documentation. Thanks very much. Best, Anthony Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.camailto:apet...@uottawa.ca -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Statistics reset after upgrade to 4.0
Hi Hilton, Thanks for the link. I actually read that well before our upgrade, but I’m not seeing anything that would cause our stats to disappear. As far as I can tell, DSpace 1.8.2 uses Solr for statistics, so all the data should be there in the database. The only that that I can see that’s changed is that our Solr server used to be at http://localhost:8080/solr and now it’s at http://localhost/solr. Not sure if that makes a difference. Anthony From: Hilton Gibson [mailto:hilton.gib...@gmail.com] Sent: Friday, July 04, 2014 9:43 AM To: Anthony Petryk Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Statistics reset after upgrade to 4.0 Hi Anthony There are some big changes regarding statistics in DSpace since version 1.8.2. See: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Statistics for a good starting point. Cheers hg Hilton Gibson Ubuntu Linux Systems Administrator JS Gericke Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Tel: +27 21 808 4100 | Cell: +27 84 646 4758 http://scholar.sun.ac.za http://bit.ly/goodir http://library.sun.ac.za http://za.linkedin.com/in/hiltongibson On 4 July 2014 15:37, Anthony Petryk anthony.pet...@uottawa.camailto:anthony.pet...@uottawa.ca wrote: Hello, After upgrading from 1.8.2 to 4.0 (via 3.2), we’ve noticed that our item, collection, etc. statistics have been reset: they’re now counting up from 0, where we previously had about a year’s worth of usage. Is there anything special that needs to be done during the upgrade to retain this data? I couldn’t find anything in the 4.0 documentation. Thanks very much. Best, Anthony Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.camailto:apet...@uottawa.ca -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Statistics reset after upgrade to 4.0
Hi Anja et al, I found the problem: when upgrading, we created a new dspace home directory and didn't bring the solr/statistics/data folder over from the old one. I just copied it over manually, optimized, and it seems to have worked. Is this the right approach, or is there a better way? Thanks everyone for your help. Anthony -Original Message- From: Anja Le Blanc [mailto:anja.lebl...@manchester.ac.uk] Sent: Friday, July 04, 2014 10:28 AM To: Anthony Petryk Subject: Re: [Dspace-tech] Statistics reset after upgrade to 4.0 Hi Anthony Did you run wget http://localhost:8080/solr/statistics/update?optimize=true DSpace 4 uses a different version of Solr (lucene). Best regards, Anja -- Anja Le Blanc Jorum Team Mimas On 04/07/2014 14:37, Anthony Petryk wrote: Hello, After upgrading from 1.8.2 to 4.0 (via 3.2), we've noticed that our item, collection, etc. statistics have been reset: they're now counting up from 0, where we previously had about a year's worth of usage. Is there anything special that needs to be done during the upgrade to retain this data? I couldn't find anything in the 4.0 documentation. Thanks very much. Best, Anthony *Anthony Petryk* Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.ca mailto:apet...@uottawa.ca -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Publication Date Metadata
Hello,
We've encountered the error reported in Sebastien's point (2) below. It occurs
when a reviewer tries to preview a workflow task. If the
webui.itemdisplay.default config line includes a date field but no value was
entered in the submission form, ItemTag.java fails when trying to display the
date.
The error won't happen if you set the given date field to required in the
submission form because the submitter must enter a date to get it out of
his/her workspace. I can confirm that dc.date.issued is not required nor set
by default. We use dc.date.created.
Here's the workaround that I used (around line 540 of ItemTag.java in the 4.0
code):
else if (isDate)
{
DCDate dd = new DCDate(values[j].value);
// Parse the date
if
(!dd.toString().equals(null)) {
out.print(UIUtil.displayDate(dd, false, false,
(HttpServletRequest)pageContext.getRequest()));
}
}
Note sure if that's the best solution but it works for us. I hope this helps.
Anthony
-Original Message-
From: Sebastien Ival [mailto:sebastien.i...@rmc.ca]
Sent: Friday, June 27, 2014 10:47 AM
To: Tim Donohue; dspace-tech@lists.sourceforge.net
Cc: Trevor Wilson
Subject: Re: [Dspace-tech] Publication Date Metadata
Hi Tim,
The default setup is what we are looking at. Users have submit theses =
original work. So the users should not have to set this date. It should be
automated as per version 3.2.
Once we did the upgrade from 3.2 to 4.1, submitting a new thesis would result
in a blank dc.date.issued.
I was trying to revert for this portion to the 3.2 behavior. It seems to me
that adding the initial questions does not fix this.
Are you trying to set dc.date.issued to today's date at all times?
Yes, this is the behavior we would like to see.
What is the reason why you'd rather your submitters not be required to
specify the dc.date.issued themselves (which is the default setting in
DSpace 4.x)?
They are submitting original work.
You did not assume wrongly. I am trying (unsuccessfully) to revert to the 3.2
behaviour.
Thanks for your help.
This is really appreciated.
-Original Message-
From: Tim Donohue [mailto:tdono...@duraspace.org]
Sent: June-27-14 10:37 AM
To: Sebastien Ival; dspace-tech@lists.sourceforge.net
Cc: Trevor Wilson
Subject: Re: [Dspace-tech] Publication Date Metadata
Hi Sebastien and Trevor,
Maybe I've misunderstood things. It might help to take a step back. What is the
setup you are trying to achieve in the DSpace submission steps?
In other words, what is your use case here?
As you mentioned, it's very possible the Initial Question step doesn't meet
your use case. If so, it might be good to clarify how you want the submission
forms to act, e.g.
Are you trying to set dc.date.issued to today's date at all times?
What is the reason why you'd rather your submitters not be required to specify
the dc.date.issued themselves (which is the default setting in DSpace 4.x)?
It's harder for us to debug this issue without a better understanding of what
it is you are trying to achieve. It sounds like I had wrongly assumed you just
wanted the same setup as in DSpace 3 (which is why I recommended just
re-enabling Initial Questions).
- Tim
On 6/27/2014 9:14 AM, Sebastien Ival wrote:
Hi Tim,
I have two points to make
1) I think as mentioned in https://jira.duraspace.org/browse/DS-1655, the
initial question step is a bit unclear and is now handled as part of the
submission process. So having to re-enable it feels like going backward. I
don't think the two should be hooked together.
2) To have the question The item has been published or publicly distributed
before showing up, I also need to add dc.identifier.citation and
dc.publisher. The field only shows up, if you answer yes to the question and
you can set all of these values otherwise, it still look like the date is not
being set.
As when I try to take the task I get an internal server error with :
-- Method: POST
-- Parameters were:
-- submit_claim: Take Task
-- workflow_id: 186
-- step: 0
org.apache.jasper.JasperException: java.lang.NullPointerException
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
at org.dspace.content.DCDate.getMonthUTC(DCDate.java:368)
at org.dspace.content.DCDate.displayUTCDate(DCDate.java:520)
at org.dspace.content.DCDate.displayDate(DCDate.java:487)
at org.dspace.app.webui.util.UIUtil.displayDate(UIUtil.java:249)
at org.dspace.app.webui.jsptag.ItemTag.render(ItemTag.java:541)
at org.dspace.app.webui.jsptag.ItemTag.doStartTag(ItemTag.java:289)
at
Re: [Dspace-tech] LAC e-theses harvesting of DSpace 3.1
Hi Graham, Try adding the oai-identifier element to [dspace-src]/dspace/config/crosswalks/oai/description.xml. Then rebuild. If you don’t see the changes, clear the oai cache or just change “cache.enable” to false in [dspace-src]/dspace/config/modules/oai.cfg. Note that I’ve only tried this on our development server (we’ll be going live with 4.0 in early July). For the identifier element, you’ll need to create a new metadata format based on the existing “etdms”. See page 350 of the Dspace 3.x documentation (section 6.15.2). I hope this helps. Best, Anthony Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.camailto:apet...@uottawa.ca From: Graham Faulkner [mailto:graham.faulk...@uwaterloo.ca] Sent: Tuesday, June 10, 2014 9:01 AM To: Terry Brady Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] LAC e-theses harvesting of DSpace 3.1 Thanks for the feedback Terry. In reviewing the LAC guidelines at www.collectionscanada.gc.ca/thesescanada/027007-9200-e.htmlhttp://www.collectionscanada.gc.ca/thesescanada/027007-9200-e.html, it looks like I’ll have to add an oai-identifier tag to the output (Section 2.5) and update the URL identifier (Section 2.7). Do you know where in the source code I should make these changes? Cheers, Graham From: Terry Brady [mailto:tw...@georgetown.edu] Sent: Monday, June 09, 2014 8:47 PM To: Graham Faulkner Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] LAC e-theses harvesting of DSpace 3.1 Since upgrading the DSpace 3.x, I have found the OAI service much easier to troubleshoot by navigating the browser-friendly OAI service. //yourrepository/oai/request?verb=Identify Does your collection/item metadata look good through that service? I believe that the naming conventions for collection and community sets may have changed when I performed the upgrade to 3.x. Terry On Mon, Jun 9, 2014 at 11:28 AM, Graham Faulkner graham.faulk...@uwaterloo.camailto:graham.faulk...@uwaterloo.ca wrote: Hi all, I’m wondering if there is someone on the list who is running DSpace 3.1+ and has a collection being properly harvested by Library and Archives Canada’s ETD harvesting program. We recently upgraded from DSpace 1.8 to 3.1 and the LAC harvesting apparently stopped working. Any pointers in the right direction are greatly appreciated. Cheers, Graham - Graham Faulkner University of Waterloo Library -- HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing Easy Data Exploration http://p.sf.net/sfu/hpccsystems ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Terry Brady Applications Programmer Analyst Georgetown University Library Information Technology https://www.library.georgetown.edu/lit/code 425-298-5498 -- HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing Easy Data Exploration http://p.sf.net/sfu/hpccsystems___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Communities and Collections in search results
Thanks for this - very helpful. Anthony -Original Message- From: zuki.ebe...@gmail.com [mailto:zuki.ebe...@gmail.com] On Behalf Of SUZUKI Keiji Sent: Sunday, April 20, 2014 2:34 AM To: Anthony Petryk Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Communities and Collections in search results Hi Anthony, 2014-04-08 4:03 GMT+09:00 Anthony Petryk anthony.pet...@uottawa.ca: In DSpace 4.0 JSPUI, the search results page includes “hits” from Communities and Collections in addition to Items. However, some of these Communities and Collections only appear on the second and third page of results (possibly because they’re ranked lower in relevance than some Items). Is there a way of getting them all to float to the top? Is it possible to remove them from the results list altogether? You can remove them with setting the following property element to default configuration settings for discovery and Homepage specific configuration settings for discovery.in [dspace]/config/spring/api/discovery.xml In the default configuration setting this property is set as comment already. property name=defaultFilterQueries list valuesearch.resourcetype:2/value /list /property This setting removes communities and collections results in XMLUI but there is a bug in JSPUI and does not remove them. I made a PR and created a ticket in JIRA. Please refer to the following URL, https://jira.duraspace.org/browse/DS-1974 Also, is it possible to configure which metadata fields are indexed for Communities and Collections? It seems that the License field is indexed, which we don’t want. You can omit the license data with setting dc.rights.license as a value of property name=toIgnoreMetadataFields also in [dspace]/config/spring/api/discovery.xml and reindex discovery index. The following is a diff with these two setting. --- discovery.xml.org 2014-04-19 15:55:58.082480988 +0900 +++ discovery.xml 2014-04-19 19:06:37.579309730 +0900 @@ -76,6 +76,8 @@ !--valuedc.description.tableofcontents/value-- !--Copyright text-- valuedc.rights/value +!--License text-- +valuedc.rights.license/value !--Collection name-- !--valuedc.title/value-- /list @@ -124,12 +126,12 @@ /bean /property !--Any default filter queries, these filter queries will be used for all queries done by discovery for this configuration-- -!--property name=defaultFilterQueries-- -!--list-- +property name=defaultFilterQueries +list !--Only find items-- -!--valuesearch.resourcetype:2/value-- -!--/list-- -!--/property-- +valuesearch.resourcetype:2/value +/list +/property !--The configuration for the recent submissions-- property name=recentSubmissionConfiguration bean class=org.dspace.discovery.configuration.DiscoveryRecentSubmissionsConfiguration @@ -212,6 +214,12 @@ ref bean=searchFilterIssued / /list /property +property name=defaultFilterQueries +list +!--Only find items-- +valuesearch.resourcetype:2/value +/list +/property !--The sort filters for the discovery search (same as defaultConfiguration above)-- property name=searchSortConfiguration bean class=org.dspace.discovery.configuration.DiscoverySortConfiguration Regards, Keiji Suzuki Ebetsu, Japan -- Start Your Social Network Today - Download eXo Platform Build your Enterprise Intranet with eXo Platform Software Java Based Open Source Intranet - Social, Extensible, Cloud Ready Get Started Now And Turn Your Intranet Into A Collaboration Platform http://p.sf.net/sfu/ExoPlatform ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Communities and Collections in search results
Hello, In DSpace 4.0 JSPUI, the search results page includes hits from Communities and Collections in addition to Items. However, some of these Communities and Collections only appear on the second and third page of results (possibly because they're ranked lower in relevance than some Items). Is there a way of getting them all to float to the top? Is it possible to remove them from the results list altogether? Also, is it possible to configure which metadata fields are indexed for Communities and Collections? It seems that the License field is indexed, which we don't want. Thanks, Anthony Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.camailto:apet...@uottawa.ca -- Put Bad Developers to Shame Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test Deployment Start a new project now. Try Jenkins in the cloud. http://p.sf.net/sfu/13600_Cloudbees___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Discovery facet configration for date
Hello, Is it possible to replace the date issued discovery facet with another field, say date created? We're using DSpace 4.0 JSPUI. Thanks. Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.camailto:apet...@uottawa.ca -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Error after upgrading from 3.2 to 4.0
Hello, I'm getting the following error (below) after upgrading DSpace from version 3.2 to 4.0. I see a similar error when trying to access the xmlui. I followed all the steps written in the manual. Note that I successfully upgrade 1.8.x to 3.2 a few days ago. Thanks in advance for your help. Anthony Petryk Emerging Technologies Librarian | Bibliothécaire des technologies émergentes uOttawa Library | Bibliothèque uOttawa 613-562-5800 x4650 apet...@uottawa.camailto:apet...@uottawa.ca -- 2014-02-06 11:44:38,259 ERROR org.dspace.jsp @ caught exception: org.dspace.plugin.PluginException: ERROR: relation community does not exist Position: 15 at org.dspace.app.webui.components.TopCommunitiesSiteProcessor.process(TopCommunitiesSiteProcessor.java:53) at org.apache.jsp.index_jsp._jspService(index_jsp.java:120) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.postgresql.util.PSQLException: ERROR: relation community does not exist Position: 15 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at org.dspace.storage.rdbms.DatabaseManager.queryTable(DatabaseManager.java:221) at org.dspace.content.Community.findAllTop(Community.java:314
Re: [Dspace-tech] Error after upgrading from 3.2 to 4.0
My mistake - I was pointing to the wrong database (I guess it helps to follow the instructions and edit build.properties!). Thanks for your help. Anthony -Original Message- From: ivan.ma...@gmail.com [mailto:ivan.ma...@gmail.com] On Behalf Of helix84 Sent: Thursday, February 06, 2014 12:32 PM To: Anthony Petryk Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after upgrading from 3.2 to 4.0 On Thu, Feb 6, 2014 at 5:54 PM, Anthony Petryk anthony.pet...@uottawa.ca wrote: org.dspace.plugin.PluginException: ERROR: relation community does not exist That means that DSpace cannot see the community table. Did you configure DSpace to use the right database and schema? Perhaps you have an empty database without tables lying around and you mistakenly told DSpace to use that. Try connecting to the same database that DSpace is configured to connect to and send me the list of tables. (the \dt command in psql). There should be around 50 tables there. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Error when adding a new bitstream format
Hello, We're getting the following error when attempting to add a new bitstream format: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint bitstreamformatregistry_pkey I checked BitstreamFormat.java and DatabaseManager.java and couldn't see any obvious cause. I also verified the bitstreamformatregistry table in the database: it has 73 rows, all sequentially numbered in the bitstream_format_id column. Is there a fix for this? I could update the database manually but would prefer that it worked via the web UI. We're on version 1.8.2. Incidentally, is there any good reason not to add zip archive to the bitstream format registry? Thanks, Anthony -- Sponsored by Intel(R) XDK Develop, test and display web and hybrid apps with a single code base. Download it for free now! http://pubads.g.doubleclick.net/gampad/clk?id=111408631iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Error when adding a new bitstream format
Perfect - thanks very much. Anthony -Original Message- From: ivan.ma...@gmail.com [mailto:ivan.ma...@gmail.com] On Behalf Of helix84 Sent: Tuesday, December 10, 2013 10:32 AM To: Anthony Petryk Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error when adding a new bitstream format On Tue, Dec 10, 2013 at 3:52 PM, Anthony Petryk anthony.pet...@uottawa.ca wrote: Is there a fix for this? I could update the database manually but would prefer that it worked via the web UI. We’re on version 1.8.2. Hi Anthony, run [dspace]/etc/postgres/update-sequences.sql (using the psql command). That will update the value of the bitstreamformatregistry_seq sequence to the highest value of bitstream_format_id in bitstreamformatregistry and thus the sequence will return an available value on the next nextval() call when adding a new row to that table. Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Rapidly troubleshoot problems before they affect your business. Most IT organizations don't have a clear picture of how application performance affects their revenue. With AppDynamics, you get 100% visibility into your Java,.NET, PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! http://pubads.g.doubleclick.net/gampad/clk?id=84349831iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Error in DSpace logs: ClientAbortException: java.net.SocketException: Broken pipe
Hello, The following error (copied below) is showing up in our DSpace logs every 2 minutes or so. I'm not sure what's causing it. Is this something to worry about? I haven't noticed any problems in the UI, but our users may have. Thanks for your help. Anthony --- 2013-01-22 10:58:32,310 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request! ClientAbortException: java.net.SocketException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288) at org.apache.catalina.connector.Response.flushBuffer(Response.java:548) at org.apache.catalina.connector.ResponseFacade.flushBuffer(ResponseFacade.java:279) at javax.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:166) at org.apache.cocoon.servletservice.HttpServletResponseBufferingWrapper.flushBufferedResponse(HttpServletResponseBufferingWrapper.java:240) at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:481) at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:443) at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:264) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy20.service(Unknown Source) at org.apache.cocoon.servletservice.DispatcherServlet.service(DispatcherServlet.java:106) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:269) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.cocoon.servlet.multipart.MultipartFilter.doFilter(MultipartFilter.java:131) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:619) Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at
Re: [Dspace-tech] Error in DSpace logs: ClientAbortException: java.net.SocketException: Broken pipe
That's a relief. :) Thanks for the quick reply. -Original Message- From: ivan.ma...@gmail.com [mailto:ivan.ma...@gmail.com] On Behalf Of helix84 Sent: Wednesday, January 23, 2013 11:06 AM To: Anthony Petryk Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error in DSpace logs: ClientAbortException: java.net.SocketException: Broken pipe On Wed, Jan 23, 2013 at 4:28 PM, Anthony Petryk anthony.pet...@uottawa.ca wrote: The following error (copied below) is showing up in our DSpace logs every 2 minutes or so. I’m not sure what’s causing it. Is this something to worry about? I haven’t noticed any problems in the UI, but our users may have. Hi Anthony, nothing to worry about. I explained it here: https://jira.duraspace.org/browse/DS-1448 Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. ON SALE this month only -- learn more at: http://p.sf.net/sfu/learnnow-d2d ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Harvest metadata and references - requires ORE support?
Hello, We're trying to harvest metadata from Hindawi into DSpace via OAI-PMH: http://www.hindawi.com/oai-pmh/. Although the harvest works successfully, the harvested items do not contain a (clickable) link back to the content. Because Hindawi does not support ORE, we cannot select Harvest metadata and references to bitstreams in the harvesting options for the collection. However, the harvested metadata does in fact contain the necessary reference, in the dc.identifier field. See http://www.hindawi.com/oai-pmh/oai.aspx?verb=ListRecordsset=ISRN.ZOOLOGY:2011metadataPrefix=oai_dc for an example. What's the best way to get the DSpace harvester to map this reference to the dc.identifier.uri field? Or is there a reason that this actually requires ORE support from the provider? We're using DSpace 1.8.2 on a test server. Thanks in advance for your insights. Anthony -- Monitor your physical, virtual and cloud infrastructure from a single web console. Get in-depth insight into apps, servers, databases, vmware, SAP, cloud infrastructure, etc. Download 30-day Free Trial. Pricing starts from $795 for 25 servers or applications! http://p.sf.net/sfu/zoho_dev2dev_nov___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
