[Dspace-tech] Error when importing statistics to elasticsearch in DSpace 5.1
Dear all, I am trying to import solr-format converted log files into Elastic Search statistics but I'm having this error when running dspace stats-log-importer-elasticsearch -i D:\converted_logs\new_dspace.log.2015 -m -s -v: Exception: None of the configured nodes are available: [] org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:278) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:197) at org.elasticsearch.client.transport.support.InternalTransportIndicesAdminClient.execute(InternalTransportIndicesAdminClient.java:86) at org.elasticsearch.client.support.AbstractIndicesAdminClient.refresh(AbstractIndicesAdminClient.java:418) at org.elasticsearch.action.admin.indices.refresh.RefreshRequestBuilder.doExecute(RefreshRequestBuilder.java:48) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65) at org.dspace.statistics.util.StatisticsImporterElasticSearch.main(StatisticsImporterElasticSearch.java:380) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:225) at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:77) I am using windows 7 but also encountered this error in windows server 2003. I have no problems when I imported my converted log files into solr statistics using the command dspace stats-log-importer. I also followed the steps to enable and configure Elasticsearch mentioned here: Elasticsearch Usage Statistics https://wiki.duraspace.org/display/DSDOC5x/Elasticsearch+Usage+Statistics . I can't find in the documentation about the settings in elastic-search-statistics.cfg. I tried to uncomment all the properties there and allowed port 9300 in my firewall. I also noticed that when I enabled Elasticsearch, the server is not loading anything. Below is portion of my dspace.log when I enabled Elasticsearch. The log just stopped there. 2015-04-07 17:12:38,537 INFO org.dspace.core.ConfigurationManager @ Loading from classloader: file:/C:/DSpace/config/dspace.cfg 2015-04-07 17:12:38,572 INFO org.dspace.core.ConfigurationManager @ Using dspace provided log configuration (log.init.config) 2015-04-07 17:12:38,574 INFO org.dspace.core.ConfigurationManager @ Loading: C:/DSpace/config/log4j.properties 2015-04-07 17:12:58,332 INFO org.dspace.statistics.util.StatisticsImporterElasticSearch @ Getting ElasticSearch Transport Client for StatisticsImporterElasticSearch... 2015-04-07 17:12:58,336 ERROR org.dspace.statistics.ElasticSearchLogger @ getClient reports null client 2015-04-07 17:12:58,339 INFO org.dspace.statistics.ElasticSearchLogger @ Creating TransportClient to [Address:127.0.0.1] [Port:9300] [cluster.name:dspacestatslogging] 2015-04-07 17:12:58,653 INFO net.sf.ehcache.util.UpdateChecker @ New update(s) found: 2.4.7 [http://www.terracotta.org/confluence/display/release/Release+Notes+Ehcache+Core+2.4] 2015-04-07 17:13:04,252 INFO org.elasticsearch.plugins @ [Buzz] loaded [], sites [] Any advice please? Thanks in advance, euler -- View this message in context: http://dspace.2283337.n4.nabble.com/Error-when-importing-statistics-to-elasticsearch-in-DSpace-5-1-tp4677334.html Sent from the DSpace - Tech mailing list archive at Nabble.com. -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] OAI error - no matches for the query
Hi All,
I got error message on my OAI testing, when I click on Identify tab on OAI
link, I get no error, when I click on SETS, no error, when I click on records I
get no matches for the query error message even when I click on Identifiers
tab I get error message, I then edited oai.cfg file as follows: still to
restart Tomcat
#---#
#XOAI CONFIGURATIONS#
#---#
# These configs are used by the XOAI#
#---#
# Storage: solr | database
storage=solr
# Base solr index
solr.url=http://localhost/solr/oai
# OAI persistent identifier prefix.
# Format - oai:PREFIX:HANDLE
identifier.prefix = localhost
# Base url for bitstreams
bitstream.baseUrl = http://myurl.wits.ac.za/
# Base Configuration Directory
config.dir = /dspace-install/config/crosswalks/oai
# Description
description.file = /dspace-install/config/crosswalks/oai/description.xml
# Cache enabled?
cache.enabled = true
# Base Cache Directory
cache.dir = /dspace/install/dspace3.2/var/oai
#---#
#--OAI HARVESTING CONFIGURATIONS#
#---#
# These configs are only used by the OAI-ORE related functions #
#---#
### Harvester settings
# Crosswalk settings; the {name} value must correspond to a declated ingestion
crosswalk
# harvester.oai.metadataformats.{name} = {namespace},{optional display name}
# The display name is only used in the xmlui for the jspui there are entries in
the
# Messages.properties in the form
jsp.tools.edit-collection.form.label21.select.{name}
harvester.oai.metadataformats.dc = http://www.openarchives.org/OAI/2.0/oai_dc/,
Simple Dublin Core
harvester.oai.metadataformats.qdc = http://purl.org/dc/terms/, Qualified Dublin
Core
harvester.oai.metadataformats.dim = http://www.dspace.org/xmlns/dspace/dim,
DSpace Intermediate Metadata
# This field works in much the same way as
harvester.oai.metadataformats.PluginName
# The {name} must correspond to a declared ingestion crosswalk, while the
# {namespace} must be supported by the target OAI-PMH provider when harvesting
content.
# harvester.oai.oreSerializationFormat.{name} = {namespace}
# Determines whether the harvester scheduling process should be started
# automatically when the DSpace webapp is deployed.
# default: false
harvester.autoStart=true
# Amount of time subtracted from the from argument of the PMH request to account
# for the time taken to negotiate a connection. Measured in seconds. Default
value is 120.
#harvester.timePadding = 120
# How frequently the harvest scheduler checks the remote provider for updates,
# messured in minutes. The default vaule is 12 hours (or 720 minutes)
#harvester.harvestFrequency = 720
oai.cfg 106 lines, 5125 characters
Regards,
Lewatle
table width=100% border=0 cellspacing=0 cellpadding=0
style=width:100%;
tr
td align=left style=text-align:justify;font face=arial,sans-serif
size=1 color=#99span style=font-size:11px;This communication is
intended for the addressee only. It is confidential. If you have received this
communication in error, please notify us immediately and destroy the original
message. You may not copy or disseminate this communication without the
permission of the University. Only authorised signatories are competent to
enter into agreements on behalf of the University and recipients are thus
advised that the content of this message may not be legally binding on the
University and may contain the personal views and opinions of the author, which
are not necessarily the views and opinions of The University of the
Witwatersrand, Johannesburg. All agreements between the University and
outsiders are subject to South African Law unless the University agrees in
writing to the contrary. /span/font/td
/tr
/table
--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] OAI error - no matches for the query
Hi Lewatle
a “no matches for the query” is not an error message. It just says
that you oai index is empty. Since DSPACE v3 thers is a new OAI engine,
and the index, an SOLR index, must be populated periodically (a nigth
cron could be suited to most installations) with a command oai import
see:
https://wiki.duraspace.org/display/DSDOC3x/OAI+2.0+Server
and
http://wiki.lib.sun.ac.za/index.php/SUNScholar/Harvesting/3.X
best regards
Emilio
El 07/04/2015 a las 11:53, Lewatle Phaladi escribió:
Hi All,
I got error message on my OAI testing, when I click on Identify tab on
OAI link, I get no error, when I click on SETS, no error, when I click
on records I get “no matches for the query” error message even when I
click on Identifiers tab I get error message, I then edited oai.cfg
file as follows: still to restart Tomcat
#---#
#XOAI CONFIGURATIONS#
#---#
# These configs are used by the XOAI#
#---#
# Storage: solr | database
storage=solr
# Base solr index
solr.url=http://localhost/solr/oai
# OAI persistent identifier prefix.
# Format - oai:PREFIX:HANDLE
identifier.prefix = localhost
# Base url for bitstreams
bitstream.baseUrl = http://myurl.wits.ac.za/
# Base Configuration Directory
config.dir = /dspace-install/config/crosswalks/oai
# Description
description.file = /dspace-install/config/crosswalks/oai/description.xml
# Cache enabled?
cache.enabled = true
# Base Cache Directory
cache.dir = /dspace/install/dspace3.2/var/oai
#---#
#--OAI HARVESTING CONFIGURATIONS#
#---#
# These configs are only used by the OAI-ORE related functions #
#---#
### Harvester settings
# Crosswalk settings; the {name} value must correspond to a declated
ingestion crosswalk
# harvester.oai.metadataformats.{name} = {namespace},{optional display
name}
# The display name is only used in the xmlui for the jspui there are
entries in the
# Messages.properties in the form
jsp.tools.edit-collection.form.label21.select.{name}
harvester.oai.metadataformats.dc =
http://www.openarchives.org/OAI/2.0/oai_dc/, Simple Dublin Core
harvester.oai.metadataformats.qdc = http://purl.org/dc/terms/,
Qualified Dublin Core
harvester.oai.metadataformats.dim =
http://www.dspace.org/xmlns/dspace/dim, DSpace Intermediate Metadata
# This field works in much the same way as
harvester.oai.metadataformats.PluginName
# The {name} must correspond to a declared ingestion crosswalk, while the
# {namespace} must be supported by the target OAI-PMH provider when
harvesting content.
# harvester.oai.oreSerializationFormat.{name} = {namespace}
# Determines whether the harvester scheduling process should be started
# automatically when the DSpace webapp is deployed.
# default: false
harvester.autoStart=true
# Amount of time subtracted from the from argument of the PMH request
to account
# for the time taken to negotiate a connection. Measured in seconds.
Default value is 120.
#harvester.timePadding = 120
# How frequently the harvest scheduler checks the remote provider for
updates,
# messured in minutes. The default vaule is 12 hours (or 720 minutes)
#harvester.harvestFrequency = 720
oai.cfg 106 lines, 5125 characters
Regards,
Lewatle
This communication is intended for the addressee only. It is
confidential. If you have received this communication in error, please
notify us immediately and destroy the original message. You may not
copy or disseminate this communication without the permission of the
University. Only authorised signatories are competent to enter into
agreements on behalf of the University and recipients are thus advised
that the content of this message may not be legally binding on the
University and may contain the personal views and opinions of the
author, which are not necessarily the views and opinions of The
University of the Witwatersrand, Johannesburg. All agreements between
the University and outsiders are subject to South African Law unless
the University agrees in writing to the contrary.
--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF
___
DSpace-tech mailing list
Re: [Dspace-tech] OAI error - no matches for the query
Hi Emilio,
Thanks for sharing, is there any tomcat restart needed after this command been
performed?
Regards,
Lewatle
From: emilio lorenzo [mailto:elore...@arvo.es]
Sent: 07 April 2015 12:48 PM
To: dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] OAI error - no matches for the query
Hi Lewatle
a no matches for the query is not an error message. It just says that you
oai index is empty. Since DSPACE v3 thers is a new OAI engine, and the index,
an SOLR index, must be populated periodically (a nigth cron could be suited to
most installations) with a command oai import
see:
https://wiki.duraspace.org/display/DSDOC3x/OAI+2.0+Server
and
http://wiki.lib.sun.ac.za/index.php/SUNScholar/Harvesting/3.X
best regards
Emilio
El 07/04/2015 a las 11:53, Lewatle Phaladi escribió:
Hi All,
I got error message on my OAI testing, when I click on Identify tab on OAI
link, I get no error, when I click on SETS, no error, when I click on records I
get no matches for the query error message even when I click on Identifiers
tab I get error message, I then edited oai.cfg file as follows: still to
restart Tomcat
#---#
#XOAI CONFIGURATIONS#
#---#
# These configs are used by the XOAI#
#---#
# Storage: solr | database
storage=solr
# Base solr index
solr.url=http://localhost/solr/oai
# OAI persistent identifier prefix.
# Format - oai:PREFIX:HANDLE
identifier.prefix = localhost
# Base url for bitstreams
bitstream.baseUrl = http://myurl.wits.ac.za/
# Base Configuration Directory
config.dir = /dspace-install/config/crosswalks/oai
# Description
description.file = /dspace-install/config/crosswalks/oai/description.xml
# Cache enabled?
cache.enabled = true
# Base Cache Directory
cache.dir = /dspace/install/dspace3.2/var/oai
#---#
#--OAI HARVESTING CONFIGURATIONS#
#---#
# These configs are only used by the OAI-ORE related functions #
#---#
### Harvester settings
# Crosswalk settings; the {name} value must correspond to a declated ingestion
crosswalk
# harvester.oai.metadataformats.{name} = {namespace},{optional display name}
# The display name is only used in the xmlui for the jspui there are entries in
the
# Messages.properties in the form
jsp.tools.edit-collection.form.label21.select.{name}
harvester.oai.metadataformats.dc = http://www.openarchives.org/OAI/2.0/oai_dc/,
Simple Dublin Core
harvester.oai.metadataformats.qdc = http://purl.org/dc/terms/, Qualified Dublin
Core
harvester.oai.metadataformats.dim = http://www.dspace.org/xmlns/dspace/dim,
DSpace Intermediate Metadata
# This field works in much the same way as
harvester.oai.metadataformats.PluginName
# The {name} must correspond to a declared ingestion crosswalk, while the
# {namespace} must be supported by the target OAI-PMH provider when harvesting
content.
# harvester.oai.oreSerializationFormat.{name} = {namespace}
# Determines whether the harvester scheduling process should be started
# automatically when the DSpace webapp is deployed.
# default: false
harvester.autoStart=true
# Amount of time subtracted from the from argument of the PMH request to account
# for the time taken to negotiate a connection. Measured in seconds. Default
value is 120.
#harvester.timePadding = 120
# How frequently the harvest scheduler checks the remote provider for updates,
# messured in minutes. The default vaule is 12 hours (or 720 minutes)
#harvester.harvestFrequency = 720
oai.cfg 106 lines, 5125 characters
Regards,
Lewatle
This communication is intended for the addressee only. It is confidential. If
you have received this communication in error, please notify us immediately and
destroy the original message. You may not copy or disseminate this
communication without the permission of the University. Only authorised
signatories are competent to enter into agreements on behalf of the University
and recipients are thus advised that the content of this message may not be
legally binding on the University and may contain the personal views and
opinions of the author, which are not necessarily the views and opinions of The
University of the Witwatersrand, Johannesburg. All agreements between the
University and outsiders are subject to South African Law unless the University
agrees in writing to the contrary.
--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live
Re: [Dspace-tech] OAI error - no matches for the query
Hello Lewatle,
no just run
[dspace]/bin/dspace oai clean-cache to remove the caches responses.
Hope this helps
Claudia
Am 07.04.2015 um 13:31 schrieb Lewatle Phaladi:
Hi Emilio,
Thanks for sharing, is there any tomcat restart needed after this command
been performed?
Regards,
Lewatle
From: emilio lorenzo [mailto:elore...@arvo.es]
Sent: 07 April 2015 12:48 PM
To: dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] OAI error - no matches for the query
Hi Lewatle
a no matches for the query is not an error message. It just says that you
oai index is empty. Since DSPACE v3 thers is a new OAI engine, and the index,
an SOLR index, must be populated periodically (a nigth cron could be suited
to most installations) with a command oai import
see:
https://wiki.duraspace.org/display/DSDOC3x/OAI+2.0+Server
and
http://wiki.lib.sun.ac.za/index.php/SUNScholar/Harvesting/3.X
best regards
Emilio
El 07/04/2015 a las 11:53, Lewatle Phaladi escribió:
Hi All,
I got error message on my OAI testing, when I click on Identify tab on OAI
link, I get no error, when I click on SETS, no error, when I click on records
I get no matches for the query error message even when I click on
Identifiers tab I get error message, I then edited oai.cfg file as follows:
still to restart Tomcat
#---#
#XOAI CONFIGURATIONS#
#---#
# These configs are used by the XOAI#
#---#
# Storage: solr | database
storage=solr
# Base solr index
solr.url=http://localhost/solr/oai
# OAI persistent identifier prefix.
# Format - oai:PREFIX:HANDLE
identifier.prefix = localhost
# Base url for bitstreams
bitstream.baseUrl = http://myurl.wits.ac.za/
# Base Configuration Directory
config.dir = /dspace-install/config/crosswalks/oai
# Description
description.file = /dspace-install/config/crosswalks/oai/description.xml
# Cache enabled?
cache.enabled = true
# Base Cache Directory
cache.dir = /dspace/install/dspace3.2/var/oai
#---#
#--OAI HARVESTING CONFIGURATIONS#
#---#
# These configs are only used by the OAI-ORE related functions #
#---#
### Harvester settings
# Crosswalk settings; the {name} value must correspond to a declated
ingestion crosswalk
# harvester.oai.metadataformats.{name} = {namespace},{optional display name}
# The display name is only used in the xmlui for the jspui there are entries
in the
# Messages.properties in the form
jsp.tools.edit-collection.form.label21.select.{name}
harvester.oai.metadataformats.dc =
http://www.openarchives.org/OAI/2.0/oai_dc/, Simple Dublin Core
harvester.oai.metadataformats.qdc = http://purl.org/dc/terms/, Qualified
Dublin Core
harvester.oai.metadataformats.dim = http://www.dspace.org/xmlns/dspace/dim,
DSpace Intermediate Metadata
# This field works in much the same way as
harvester.oai.metadataformats.PluginName
# The {name} must correspond to a declared ingestion crosswalk, while the
# {namespace} must be supported by the target OAI-PMH provider when
harvesting content.
# harvester.oai.oreSerializationFormat.{name} = {namespace}
# Determines whether the harvester scheduling process should be started
# automatically when the DSpace webapp is deployed.
# default: false
harvester.autoStart=true
# Amount of time subtracted from the from argument of the PMH request to
account
# for the time taken to negotiate a connection. Measured in seconds. Default
value is 120.
#harvester.timePadding = 120
# How frequently the harvest scheduler checks the remote provider for updates,
# messured in minutes. The default vaule is 12 hours (or 720 minutes)
#harvester.harvestFrequency = 720
oai.cfg 106 lines, 5125 characters
Regards,
Lewatle
This communication is intended for the addressee only. It is confidential. If
you have received this communication in error, please notify us immediately
and destroy the original message. You may not copy or disseminate this
communication without the permission of the University. Only authorised
signatories are competent to enter into agreements on behalf of the
University and recipients are thus advised that the content of this message
may not be legally binding on the University and may contain the personal
views and opinions of the author, which are not necessarily the views and
opinions of The University of the Witwatersrand, Johannesburg. All agreements
between the University and outsiders are subject to South African Law unless
the University agrees in writing to the contrary.
Re: [Dspace-tech] OAI error - no matches for the query
Thanks Claudia!
Regards,
Lewatle
-Original Message-
From: Claudia Jürgen [mailto:claudia.juer...@ub.tu-dortmund.de]
Sent: 07 April 2015 02:04 PM
To: dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] OAI error - no matches for the query
Hello Lewatle,
no just run
[dspace]/bin/dspace oai clean-cache to remove the caches responses.
Hope this helps
Claudia
Am 07.04.2015 um 13:31 schrieb Lewatle Phaladi:
Hi Emilio,
Thanks for sharing, is there any tomcat restart needed after this command
been performed?
Regards,
Lewatle
From: emilio lorenzo [mailto:elore...@arvo.es]
Sent: 07 April 2015 12:48 PM
To: dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] OAI error - no matches for the query
Hi Lewatle
a no matches for the query is not an error message. It just says that you
oai index is empty. Since DSPACE v3 thers is a new OAI engine, and the index,
an SOLR index, must be populated periodically (a nigth cron could be suited
to most installations) with a command oai import
see:
https://wiki.duraspace.org/display/DSDOC3x/OAI+2.0+Server
and
http://wiki.lib.sun.ac.za/index.php/SUNScholar/Harvesting/3.X
best regards
Emilio
El 07/04/2015 a las 11:53, Lewatle Phaladi escribió:
Hi All,
I got error message on my OAI testing, when I click on Identify tab on
OAI link, I get no error, when I click on SETS, no error, when I click
on records I get no matches for the query error message even when I
click on Identifiers tab I get error message, I then edited oai.cfg
file as follows: still to restart Tomcat
#---#
#XOAI CONFIGURATIONS#
#---#
# These configs are used by the XOAI#
#---#
# Storage: solr | database
storage=solr
# Base solr index
solr.url=http://localhost/solr/oai
# OAI persistent identifier prefix.
# Format - oai:PREFIX:HANDLE
identifier.prefix = localhost
# Base url for bitstreams
bitstream.baseUrl = http://myurl.wits.ac.za/
# Base Configuration Directory
config.dir = /dspace-install/config/crosswalks/oai
# Description
description.file =
/dspace-install/config/crosswalks/oai/description.xml
# Cache enabled?
cache.enabled = true
# Base Cache Directory
cache.dir = /dspace/install/dspace3.2/var/oai
#---#
#--OAI HARVESTING CONFIGURATIONS#
#---#
# These configs are only used by the OAI-ORE related functions #
#---#
### Harvester settings
# Crosswalk settings; the {name} value must correspond to a declated
ingestion crosswalk # harvester.oai.metadataformats.{name} =
{namespace},{optional display name} # The display name is only used in
the xmlui for the jspui there are entries in the # Messages.properties
in the form jsp.tools.edit-collection.form.label21.select.{name}
harvester.oai.metadataformats.dc =
http://www.openarchives.org/OAI/2.0/oai_dc/, Simple Dublin Core
harvester.oai.metadataformats.qdc = http://purl.org/dc/terms/,
Qualified Dublin Core harvester.oai.metadataformats.dim =
http://www.dspace.org/xmlns/dspace/dim, DSpace Intermediate Metadata
# This field works in much the same way as
harvester.oai.metadataformats.PluginName
# The {name} must correspond to a declared ingestion crosswalk, while
the # {namespace} must be supported by the target OAI-PMH provider when
harvesting content.
# harvester.oai.oreSerializationFormat.{name} = {namespace}
# Determines whether the harvester scheduling process should be
started # automatically when the DSpace webapp is deployed.
# default: false
harvester.autoStart=true
# Amount of time subtracted from the from argument of the PMH request
to account # for the time taken to negotiate a connection. Measured in
seconds. Default value is 120.
#harvester.timePadding = 120
# How frequently the harvest scheduler checks the remote provider for
updates, # messured in minutes. The default vaule is 12 hours (or 720
minutes) #harvester.harvestFrequency = 720 oai.cfg 106 lines, 5125
characters
Regards,
Lewatle
This communication is intended for the addressee only. It is confidential. If
you have received this communication in error, please notify us immediately
and destroy the original message. You may not copy or disseminate this
communication without the permission of the University. Only authorised
signatories are competent to enter into agreements on behalf of the
University and recipients are thus advised that the content of this message
may not be legally binding on the University and may contain the personal
views and opinions of the
Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information)
Hi George, Yes, that is what I was suggesting...upgrading directly from 1.8.2 or 3.3 (whichever was your starting point) to 5.1. The ant update should take care of the necessary Solr index upgrades for you. - Tim On 4/6/2015 6:21 PM, George Stanley Kozak wrote: Tim: Thank you for your advice. I did try the 5.1 ant update, but it failed. Maybe if I tried it going from DSpace 3.3 or 1.8.2 directly to DSpace 5.1 instead of doing the intermediate step to 4.3, this might work? I will try a few other things and let everyone know what works. George Kozak Digital Library Specialist Cornell University Library Information Technologies (CULIT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 From: Tim Donohue tdono...@duraspace.org Sent: Monday, April 6, 2015 4:44 PM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, The error below is essentially saying that your Solr statistics index is too old for the version of Solr packages with DSpace 4.3 to upgrade. However, in the DSpace 5.1 upgrade process, during the ant update step, we've attempted to catch this scenario automatically -- we try to determine the version of a Solr index, and upgrade it automatically to the latest version. So, I'd recommend trying to point DSpace 5.1 at your old Solr index, and running ant update. It is supposed to upgrade an old index automatically. If that doesn't work for some reason, another option is to do a more manual upgrade, as described in the DSpace 5.x documentation. This manual upgrade essentially requires downloading multiple versions of the Solr/Lucene core JAR, and running it against your index in order to upgrade it to the latest compatible version (this is essentially the same process that is now automated by ant update though) https://wiki.duraspace.org/display/DSDOC5x/Upgrading+DSpace#UpgradingDSpace-ManuallyUpgradingSolrIndexes By the way, from your earlier messages, recently I also did encounter that odd Solr write.lock error (Index locked for write..) during a recent upgrade I tried to 5.1. In my situation, it seemed like it was a stale write.lock file which was somehow sitting around. After clearing it out, the upgrade proceeded. Admittedly, I still need to dig a bit further and ensure my analysis is correct. - Tim On 4/1/2015 3:16 PM, George Stanley Kozak wrote: Hi, Everyone: I have discovered that when I try to optimize my Solr indexes when I upgraded from DSpace 3.3 to 4.3 (wget ‘http://localhost:8080/solr/statistics/update?optimize=true’) I get the following errors in the solr logs: ERROR org.apache.solr.core.CoreContainer @ Unable to create core: statistics Format version is not supported (resource: segment _32 in resource ChecksumIndexInput(MMapIndexInput(path=/cul/app/dspace/solr/statistics/data/index/segments_1p0))): 2.x. This version of Lucene only supports indexes created with release 3.0 and later. If I later try to upgrade to 5.1, I see during the ant update: /cul/app/dspace/src/dspace-5.1-src-release/dspace/target/dspace-installer/build.xml:1061: ERROR occurred while checking Solr index version: Exception in thread main java.io.IOException: Could not read Lucene segments files in /cul/app/dspace/solr/statistics/data/index I know the 5.1 documentation states that you can manually update the Solr indexes. Should that be done in my 4.3 upgrade before I try to go to 5.1 or should it be done in the 5.1 upgrade? George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___
Re: [Dspace-tech] A group for all registered users
Hello Chris, this is configurable in the different authentification configs. If you use password authentification take a look at [dspace]/config/modules/authentification-password.cfg # Password users group # # If required, a group name can be given here, and all users who log in # using the DSpace password system will automatically become members of # this group. This is useful if you want a group made up of all internal # authenticated users. # login.specialgroup = group-name Define the group there and give the group submit right to the collection. Hope this helps Claudia Am 07.04.2015 um 17:28 schrieb Chris Gray: We are trying to set up our 5.1 instance to allow all registered users to submit to a particular collection without an administrator having to intervene for each new user. Is there a way to do this? The Business Logic document is very vague on the subject of special groups. What I'm hoping is to be able to configure a setting to automatically associate each registered user with a Registered Users group. But I begin to get a sinking feeling that I will have to hack at the eperson classes to do this. Chris -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Claudia Juergen Eldorado Technische Universität Dortmund Universitätsbibliothek Vogelpothsweg 76 44227 Dortmund Tel.: +49 231-755 40 43 Fax: +49 231-755 40 32 claudia.juer...@ub.tu-dortmund.de www.ub.tu-dortmund.de Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie ist ausschließlich für den Adressaten bestimmt. Sollten Sie nicht der für diese E-Mail bestimmte Adressat sein, unterrichten Sie bitte den Absender und vernichten Sie diese Mail. Vielen Dank. Unbeschadet der Korrespondenz per E-Mail sind unsere Erklärungen ausschließlich final rechtsverbindlich, wenn sie in herkömmlicher Schriftform (mit eigenhändiger Unterschrift) oder durch Übermittlung eines solchen Schriftstücks per Telefax erfolgen. Important note: The information included in this e-mail is confidential. It is solely intended for the recipient. If you are not the intended recipient of this e-mail please contact the sender and delete this message. Thank you. Without prejudice of e-mail correspondence our statements are only legally binding when they are made in the conventional written form (with personal signature) or when such documents are sent by fax. -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] A group for all registered users
We are trying to set up our 5.1 instance to allow all registered users to submit to a particular collection without an administrator having to intervene for each new user. Is there a way to do this? The Business Logic document is very vague on the subject of special groups. What I'm hoping is to be able to configure a setting to automatically associate each registered user with a Registered Users group. But I begin to get a sinking feeling that I will have to hack at the eperson classes to do this. Chris -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information)
Hi George, What happens if you do the following against your old 1.8.2 indexes: 1. wget http://search.maven.org/remotecontent?filepath=org/apache/lucene/lucene-core/3.5.0/lucene-core-3.5.0.jar; 2. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /cul/app/dspace/solr/statistics/data/index 3. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /cul/app/dspace/solr/search/data/index Do either of those IndexUpgrader commands succeed against your *old* 1.8.2 indexes? If not, could you send along the exception stacktrace? I'm just trying to determine if the issue is actually in these old 1.8.2 indexes. It may be either they are indexes from an unexpected, older version of Solr, or there's some minor index corruption here. -- To be honest, another thing to note here is that your *search* indexes can be easily rebuilt in DSpace 5.x (simply run ./dspace index-discovery -b). The only older index that really requires this upgrade process is the *statistics* index (as currently it doesn't have a reindex command). So, if your statistics index is of no importance, you could just delete both of these old indexes and upgrade to 5.1. As part of the 5.1 upgrade, a full reindex of the search index will be kicked off (when you start Tomcat). So, after a few minutes, your content should be fully reindexed and working. - Tim On 4/7/2015 10:12 AM, George Stanley Kozak wrote: Thanks for your help, Tim: I'm afraid that I keep hitting a wall when I try to upgrade to DSpace 5.1 I tried to go directly from DSpace 1.8.2 to DSpace 5.1 and got: ERROR occurred while checking Solr index version: Exception in thread main java.io.IOException: Could not read Lucene segments files in /cul/app/dspace/solr/statistics/data/index We have been using our own stats gathering system instead of the one within DSpace. So, I removed the statistics index but then I got: Exception in thread main java.io.IOException: Could not read Lucene segments files in /cul/app/dspace/solr/search/data/index This version of Lucene only supports indexes created with release 3.0 and later. So, I went ahead and used the DSpace 4.3 version of our solr indexes (without the statistics). It ran fine and everything got upgraded, but when I tried to do any searching, I get the write lock error. If I turn off tomcat, delete the write lock and restart things, I get the same thing. Also, I cannot update the discovery index because of the write lock problem. By the way, I did try the manual upgrade of the Solr indexes defined in the manual, but that didn't work for me, either. So, right now, it looks like I can only upgrade to DSpace 4.3, unless you have some more suggestions for me. George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -Original Message- From: Tim Donohue [mailto:tdono...@duraspace.org] Sent: Tuesday, April 07, 2015 9:31 AM To: George Stanley Kozak; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, Yes, that is what I was suggesting...upgrading directly from 1.8.2 or 3.3 (whichever was your starting point) to 5.1. The ant update should take care of the necessary Solr index upgrades for you. - Tim On 4/6/2015 6:21 PM, George Stanley Kozak wrote: Tim: Thank you for your advice. I did try the 5.1 ant update, but it failed. Maybe if I tried it going from DSpace 3.3 or 1.8.2 directly to DSpace 5.1 instead of doing the intermediate step to 4.3, this might work? I will try a few other things and let everyone know what works. George Kozak Digital Library Specialist Cornell University Library Information Technologies (CULIT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 From: Tim Donohue tdono...@duraspace.org Sent: Monday, April 6, 2015 4:44 PM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, The error below is essentially saying that your Solr statistics index is too old for the version of Solr packages with DSpace 4.3 to upgrade. However, in the DSpace 5.1 upgrade process, during the ant update step, we've attempted to catch this scenario automatically -- we try to determine the version of a Solr index, and upgrade it automatically to the latest version. So, I'd recommend trying to point DSpace 5.1 at your old Solr index, and running ant update. It is supposed to upgrade an old index automatically. If that doesn't work for some reason, another option is to do a more manual upgrade, as described in the DSpace 5.x documentation. This manual upgrade essentially requires downloading multiple versions of the Solr/Lucene core JAR, and running it
Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information)
Hi George, On 4/7/2015 1:09 PM, George Stanley Kozak wrote: Hi, Tim: 1) I did the lucene-core-3.5.0.jar manual update of the indexes. For the search index, I got no errors, but for the statistics core, I got: Exception in thread main org.apache.lucene.index.IndexNotFoundException: org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65 at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118) at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85) This would imply that the search index upgraded fine (no errors). But, the error for your statistics index almost implies that it's not a valid Solr index. Do you use the statistics index? Is there anything in the /cul/app/dspace/solr/statistics/data/index directory or is it empty? 2) I deleted the statistics core and did the ant update. I brought up my DSpace 5.1 test system and when I tried to do a search, I got no results. When I went to do a browse, I got: ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ anonymous:session_id=5749DBA88953D8B0ABDDA3FFD1DFF5F2:ip_addr=128.84.117.195:Error in Discovery while setting up date facet range:date facet\colon; dateIssued.year SolrCore 'search' is not available due to init failure: Index locked for write for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search' is not available due to init failure: Index locked for write for core search 3) I stopped tomcat, deleted the search index (as you suggested) and did the mvn build and ant update and then copied the new webapps over to tomcat. Everything looked good. I started tomcat and my site came up. Before doing anything more I ran ./dspace index-discovery -b. It ran to completion, but all searches return no hits and the browses give me the same write lock error as above. I made sure the permissions are correct (dspace user owns the solr indexes and tomcat) and removed the write lock and tried again, but again I get the same error. These other two errors seem rather odd to me...like something else is accessing Solr at the same time in which you are trying to reindex, and those multiple simultaneous writes are throwing errors. Can you try something even more simplistic than what you've done above? 1. Stop Tomcat 2. Copy your 1.8.3 data over to where you will install DSpace 5 3. Immediately delete *everything* under /cul/app/dspace/solr/ (including the entire statistics and search subdirectories). This will wipe out all existing indexes, obviously. 4. Build DSpace 5.1 (mvn package). Then run ant update. This should complete, as you'll have no existing Solr indexes for it to complain about. 5. Finally, start Tomcat back up. Now, sit back and wait. Don't run any other commands. But, feel free to check the logs for any issues as Tomcat boots up. Behind the scenes, DSpace should *automatically* be reindexing your content (and recreating the search index from nothing). You won't need to manually run anything (like 'index-discovery -b'), as this is all automated in DSpace 5 on the first bootup. (After the first bootup though, 'index-discovery -b' is the way to force a reindex.) I'm just curious if starting from *completely empty* indexes and installing DSpace 5 (without running 'index-discovery' manually) will act any different. If you still hit problems here, I'd recommend sending us a full stacktrace of the error messages you encounter. - Tim George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -Original Message- From: Tim Donohue [mailto:tdono...@duraspace.org] Sent: Tuesday, April 07, 2015 11:52 AM To: George Stanley Kozak; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, What happens if you do the following against your old 1.8.2 indexes: 1. wget http://search.maven.org/remotecontent?filepath=org/apache/lucene/lucene-core/3.5.0/lucene-core-3.5.0.jar; 2. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /cul/app/dspace/solr/statistics/data/index 3. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /cul/app/dspace/solr/search/data/index Do either of those IndexUpgrader commands succeed against your *old* 1.8.2 indexes? If not, could you send along the exception stacktrace? I'm just trying to determine if the issue is actually in these old 1.8.2 indexes. It may be either they are indexes from an unexpected, older version of Solr, or there's some minor index corruption here. -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process
Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information)
Hi, Tim: 1) I did the lucene-core-3.5.0.jar manual update of the indexes. For the search index, I got no errors, but for the statistics core, I got: Exception in thread main org.apache.lucene.index.IndexNotFoundException: org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65 at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118) at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85) 2) I deleted the statistics core and did the ant update. I brought up my DSpace 5.1 test system and when I tried to do a search, I got no results. When I went to do a browse, I got: ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ anonymous:session_id=5749DBA88953D8B0ABDDA3FFD1DFF5F2:ip_addr=128.84.117.195:Error in Discovery while setting up date facet range:date facet\colon; dateIssued.year SolrCore 'search' is not available due to init failure: Index locked for write for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search' is not available due to init failure: Index locked for write for core search 3) I stopped tomcat, deleted the search index (as you suggested) and did the mvn build and ant update and then copied the new webapps over to tomcat. Everything looked good. I started tomcat and my site came up. Before doing anything more I ran ./dspace index-discovery -b. It ran to completion, but all searches return no hits and the browses give me the same write lock error as above. I made sure the permissions are correct (dspace user owns the solr indexes and tomcat) and removed the write lock and tried again, but again I get the same error. George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -Original Message- From: Tim Donohue [mailto:tdono...@duraspace.org] Sent: Tuesday, April 07, 2015 11:52 AM To: George Stanley Kozak; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, What happens if you do the following against your old 1.8.2 indexes: 1. wget http://search.maven.org/remotecontent?filepath=org/apache/lucene/lucene-core/3.5.0/lucene-core-3.5.0.jar; 2. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /cul/app/dspace/solr/statistics/data/index 3. java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /cul/app/dspace/solr/search/data/index Do either of those IndexUpgrader commands succeed against your *old* 1.8.2 indexes? If not, could you send along the exception stacktrace? I'm just trying to determine if the issue is actually in these old 1.8.2 indexes. It may be either they are indexes from an unexpected, older version of Solr, or there's some minor index corruption here. -- To be honest, another thing to note here is that your *search* indexes can be easily rebuilt in DSpace 5.x (simply run ./dspace index-discovery -b). The only older index that really requires this upgrade process is the *statistics* index (as currently it doesn't have a reindex command). So, if your statistics index is of no importance, you could just delete both of these old indexes and upgrade to 5.1. As part of the 5.1 upgrade, a full reindex of the search index will be kicked off (when you start Tomcat). So, after a few minutes, your content should be fully reindexed and working. - Tim On 4/7/2015 10:12 AM, George Stanley Kozak wrote: Thanks for your help, Tim: I'm afraid that I keep hitting a wall when I try to upgrade to DSpace 5.1 I tried to go directly from DSpace 1.8.2 to DSpace 5.1 and got: ERROR occurred while checking Solr index version: Exception in thread main java.io.IOException: Could not read Lucene segments files in /cul/app/dspace/solr/statistics/data/index We have been using our own stats gathering system instead of the one within DSpace. So, I removed the statistics index but then I got: Exception in thread main java.io.IOException: Could not read Lucene segments files in /cul/app/dspace/solr/search/data/index This version of Lucene only supports indexes created with release 3.0 and later. So, I went ahead and used the DSpace 4.3 version of our solr indexes (without the statistics). It ran fine and everything got upgraded, but when I tried to do any searching, I get the write lock error. If I turn off tomcat, delete the write lock and restart things, I get the same thing. Also, I cannot update the discovery index because of the write lock problem. By the way, I did try the manual upgrade of the Solr indexes defined in the manual, but that didn't work for me, either. So, right now, it looks like I can only upgrade to DSpace 4.3, unless you have some more suggestions for me. George Kozak Digital Library Specialist
Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information)
Hi, Tim: To answer one of your questions: my original DSpace 1.8.2 solr statistics core did have data in it. Attached is a snapshot of the index directory. Now, I did as you asked. I deleted the all of the solr cores (actually everything under /cul/app/dspace/solr) I did the mvn and ant update and everything went well. I copied the webapps over to tomcat and restarted tomcat. I watched the DSpace log and the catalina.out log. Everything seemed to move along smoothly. When everything was done, I waited a few minutes before I went to my DSpace home page. When my home page came up, I began to see errors in my DSpace log: ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ anonymous:session_id=B3B12E105B95C8FB8E63E9FB94B5971F:ip_addr=128.84.117.195:Error in Discovery while setting up date facet range:date facet\colon; dateIssued.year SolrCore 'search' is not available due to init failure: Index locked for write for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search' is not available due to init failure: Index locked for write for core search Any attempt to do any browse or search results with errors. By the way, the errors on the web appear with org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. And then raw HTML after that. What has me stumped is that outside of the problem with the solr statistics, none of these problems appear in my DSpace 4.3 upgrade. I have no write lock problems in DSpace 4.3 install. George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -Original Message- From: Tim Donohue [mailto:tdono...@duraspace.org] Sent: Tuesday, April 07, 2015 2:33 PM To: George Stanley Kozak; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, On 4/7/2015 1:09 PM, George Stanley Kozak wrote: Hi, Tim: 1) I did the lucene-core-3.5.0.jar manual update of the indexes. For the search index, I got no errors, but for the statistics core, I got: Exception in thread main org.apache.lucene.index.IndexNotFoundException: org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65 at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118) at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85) This would imply that the search index upgraded fine (no errors). But, the error for your statistics index almost implies that it's not a valid Solr index. Do you use the statistics index? Is there anything in the /cul/app/dspace/solr/statistics/data/index directory or is it empty? 2) I deleted the statistics core and did the ant update. I brought up my DSpace 5.1 test system and when I tried to do a search, I got no results. When I went to do a browse, I got: ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ anonymous:session_id=5749DBA88953D8B0ABDDA3FFD1DFF5F2:ip_addr=128.84.1 17.195:Error in Discovery while setting up date facet range:date facet\colon; dateIssued.year SolrCore 'search' is not available due to init failure: Index locked for write for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search' is not available due to init failure: Index locked for write for core search 3) I stopped tomcat, deleted the search index (as you suggested) and did the mvn build and ant update and then copied the new webapps over to tomcat. Everything looked good. I started tomcat and my site came up. Before doing anything more I ran ./dspace index-discovery -b. It ran to completion, but all searches return no hits and the browses give me the same write lock error as above. I made sure the permissions are correct (dspace user owns the solr indexes and tomcat) and removed the write lock and tried again, but again I get the same error. These other two errors seem rather odd to me...like something else is accessing Solr at the same time in which you are trying to reindex, and those multiple simultaneous writes are throwing errors. Can you try something even more simplistic than what you've done above? 1. Stop Tomcat 2. Copy your 1.8.3 data over to where you will install DSpace 5 3. Immediately delete *everything* under /cul/app/dspace/solr/ (including the entire statistics and search subdirectories). This will wipe out all existing indexes, obviously. 4. Build DSpace 5.1 (mvn package). Then run ant update. This should complete, as you'll have no existing Solr indexes for it to complain about. 5. Finally, start Tomcat back up. Now, sit back and wait. Don't run any other commands. But, feel free to check the logs for any issues as Tomcat boots
Re: [Dspace-tech] Error when importing statistics to elasticsearch in DSpace 5.1
Hi Euler, The issue looks like this command line operation couldn't find an Elastic Search node. Is tomcat running? In the out-of-the-box architecture, DSpace will run an Elastic Search server within tomcat. So, ensure you have elasticsearch enabled, and then visit a dspace object such that a usage event is triggered. (View an item). The logs should show that an Elastic Search instance was started. Once that node is up, then you can run this data import. A more complicate architecture, that works well for sites with larger infrastructure, is to have a dedicated elastic search server, and then configure the dspace/config/elasticsearch.cfg to use that IP address. Then, the command line operation should work, regardless of the tomcat setup. Peter Dietz Longsight www.longsight.com pe...@longsight.com p: 740-599-5005 x809 On Tue, Apr 7, 2015 at 5:06 AM, euler esne...@seafdec.org.ph wrote: Dear all, I am trying to import solr-format converted log files into Elastic Search statistics but I'm having this error when running dspace stats-log-importer-elasticsearch -i D:\converted_logs\new_dspace.log.2015 -m -s -v: Exception: None of the configured nodes are available: [] org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:278) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:197) at org.elasticsearch.client.transport.support.InternalTransportIndicesAdminClient.execute(InternalTransportIndicesAdminClient.java:86) at org.elasticsearch.client.support.AbstractIndicesAdminClient.refresh(AbstractIndicesAdminClient.java:418) at org.elasticsearch.action.admin.indices.refresh.RefreshRequestBuilder.doExecute(RefreshRequestBuilder.java:48) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65) at org.dspace.statistics.util.StatisticsImporterElasticSearch.main(StatisticsImporterElasticSearch.java:380) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:225) at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:77) I am using windows 7 but also encountered this error in windows server 2003. I have no problems when I imported my converted log files into solr statistics using the command dspace stats-log-importer. I also followed the steps to enable and configure Elasticsearch mentioned here: Elasticsearch Usage Statistics https://wiki.duraspace.org/display/DSDOC5x/Elasticsearch+Usage+Statistics . I can't find in the documentation about the settings in elastic-search-statistics.cfg. I tried to uncomment all the properties there and allowed port 9300 in my firewall. I also noticed that when I enabled Elasticsearch, the server is not loading anything. Below is portion of my dspace.log when I enabled Elasticsearch. The log just stopped there. 2015-04-07 17:12:38,537 INFO org.dspace.core.ConfigurationManager @ Loading from classloader: file:/C:/DSpace/config/dspace.cfg 2015-04-07 17:12:38,572 INFO org.dspace.core.ConfigurationManager @ Using dspace provided log configuration (log.init.config) 2015-04-07 17:12:38,574 INFO org.dspace.core.ConfigurationManager @ Loading: C:/DSpace/config/log4j.properties 2015-04-07 17:12:58,332 INFO org.dspace.statistics.util.StatisticsImporterElasticSearch @ Getting ElasticSearch Transport Client for StatisticsImporterElasticSearch... 2015-04-07 17:12:58,336 ERROR org.dspace.statistics.ElasticSearchLogger @ getClient reports null client 2015-04-07 17:12:58,339 INFO org.dspace.statistics.ElasticSearchLogger @ Creating TransportClient to [Address:127.0.0.1] [Port:9300] [cluster.name:dspacestatslogging] 2015-04-07 17:12:58,653 INFO net.sf.ehcache.util.UpdateChecker @ New update(s) found: 2.4.7 [ http://www.terracotta.org/confluence/display/release/Release+Notes+Ehcache+Core+2.4 ] 2015-04-07 17:13:04,252 INFO org.elasticsearch.plugins @ [Buzz] loaded [], sites [] Any advice please? Thanks in advance, euler -- View this message in context: http://dspace.2283337.n4.nabble.com/Error-when-importing-statistics-to-elasticsearch-in-DSpace-5-1-tp4677334.html Sent from the DSpace - Tech mailing list archive at Nabble.com. -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM
Re: [Dspace-tech] A group for all registered users
Thank you, Claudia. That works. Our test server is using password authentication for now. One thing I noticed is that if the specialgroup has submit permission for a collection, that collection does not occur in the drop-down list for users. Instead, they must browse to the collection and then the submit to this collection link appears. Is there something special to get the collection into the submission drop-down list? Chris On 15-04-07 12:04 PM, Claudia Jürgen wrote: Hello Chris, this is configurable in the different authentification configs. If you use password authentification take a look at [dspace]/config/modules/authentification-password.cfg # Password users group # # If required, a group name can be given here, and all users who log in # using the DSpace password system will automatically become members of # this group. This is useful if you want a group made up of all internal # authenticated users. # login.specialgroup = group-name Define the group there and give the group submit right to the collection. Hope this helps Claudia Am 07.04.2015 um 17:28 schrieb Chris Gray: We are trying to set up our 5.1 instance to allow all registered users to submit to a particular collection without an administrator having to intervene for each new user. Is there a way to do this? The Business Logic document is very vague on the subject of special groups. What I'm hoping is to be able to configure a setting to automatically associate each registered user with a Registered Users group. But I begin to get a sinking feeling that I will have to hack at the eperson classes to do this. Chris -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information)
Good afternoon, We saw very similar errors (particularly Expected mime type application/octet-stream but got text/html) and found that we hadn't updated the appBase directory in tomcat.xml. Another symptom of ours was the code of the entire error page being written to the dspace log. It's a longshot, but perhaps something to check. B-- -Original Message- From: George Stanley Kozak [mailto:g...@cornell.edu] Sent: Tuesday, April 7, 2015 1:28 PM To: Tim Donohue; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi, Tim: To answer one of your questions: my original DSpace 1.8.2 solr statistics core did have data in it. Attached is a snapshot of the index directory. Now, I did as you asked. I deleted the all of the solr cores (actually everything under /cul/app/dspace/solr) I did the mvn and ant update and everything went well. I copied the webapps over to tomcat and restarted tomcat. I watched the DSpace log and the catalina.out log. Everything seemed to move along smoothly. When everything was done, I waited a few minutes before I went to my DSpace home page. When my home page came up, I began to see errors in my DSpace log: ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ anonymous:session_id=B3B12E105B95C8FB8E63E9FB94B5971F:ip_addr=128.84.117.195:Error in Discovery while setting up date facet range:date facet\colon; dateIssued.year SolrCore 'search' is not available due to init failure: Index locked for write for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search' is not available due to init failure: Index locked for write for core search Any attempt to do any browse or search results with errors. By the way, the errors on the web appear with org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. And then raw HTML after that. What has me stumped is that outside of the problem with the solr statistics, none of these problems appear in my DSpace 4.3 upgrade. I have no write lock problems in DSpace 4.3 install. George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -Original Message- From: Tim Donohue [mailto:tdono...@duraspace.org] Sent: Tuesday, April 07, 2015 2:33 PM To: George Stanley Kozak; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, On 4/7/2015 1:09 PM, George Stanley Kozak wrote: Hi, Tim: 1) I did the lucene-core-3.5.0.jar manual update of the indexes. For the search index, I got no errors, but for the statistics core, I got: Exception in thread main org.apache.lucene.index.IndexNotFoundException: org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65 at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118) at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85) This would imply that the search index upgraded fine (no errors). But, the error for your statistics index almost implies that it's not a valid Solr index. Do you use the statistics index? Is there anything in the /cul/app/dspace/solr/statistics/data/index directory or is it empty? 2) I deleted the statistics core and did the ant update. I brought up my DSpace 5.1 test system and when I tried to do a search, I got no results. When I went to do a browse, I got: ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ anonymous:session_id=5749DBA88953D8B0ABDDA3FFD1DFF5F2:ip_addr=128.84.1 17.195:Error in Discovery while setting up date facet range:date facet\colon; dateIssued.year SolrCore 'search' is not available due to init failure: Index locked for write for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search' is not available due to init failure: Index locked for write for core search 3) I stopped tomcat, deleted the search index (as you suggested) and did the mvn build and ant update and then copied the new webapps over to tomcat. Everything looked good. I started tomcat and my site came up. Before doing anything more I ran ./dspace index-discovery -b. It ran to completion, but all searches return no hits and the browses give me the same write lock error as above. I made sure the permissions are correct (dspace user owns the solr indexes and tomcat) and removed the write lock and tried again, but again I get the same error. These other two errors seem rather odd to me...like something else is accessing Solr at the same time in which you are trying to reindex, and those multiple simultaneous writes are throwing errors. Can you try something
Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information)
Hi, Brian: Sorry for being dense, but what do you mean by updating the appBase directory? George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -Original Message- From: Brian Freels-Stendel [mailto:bfre...@unm.edu] Sent: Tuesday, April 07, 2015 3:47 PM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Good afternoon, We saw very similar errors (particularly Expected mime type application/octet-stream but got text/html) and found that we hadn't updated the appBase directory in tomcat.xml. Another symptom of ours was the code of the entire error page being written to the dspace log. It's a longshot, but perhaps something to check. B-- -Original Message- From: George Stanley Kozak [mailto:g...@cornell.edu] Sent: Tuesday, April 7, 2015 1:28 PM To: Tim Donohue; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi, Tim: To answer one of your questions: my original DSpace 1.8.2 solr statistics core did have data in it. Attached is a snapshot of the index directory. Now, I did as you asked. I deleted the all of the solr cores (actually everything under /cul/app/dspace/solr) I did the mvn and ant update and everything went well. I copied the webapps over to tomcat and restarted tomcat. I watched the DSpace log and the catalina.out log. Everything seemed to move along smoothly. When everything was done, I waited a few minutes before I went to my DSpace home page. When my home page came up, I began to see errors in my DSpace log: ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ anonymous:session_id=B3B12E105B95C8FB8E63E9FB94B5971F:ip_addr=128.84.117.195:Error in Discovery while setting up date facet range:date facet\colon; dateIssued.year SolrCore 'search' is not available due to init failure: Index locked for write for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search' is not available due to init failure: Index locked for write for core search Any attempt to do any browse or search results with errors. By the way, the errors on the web appear with org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. And then raw HTML after that. What has me stumped is that outside of the problem with the solr statistics, none of these problems appear in my DSpace 4.3 upgrade. I have no write lock problems in DSpace 4.3 install. George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -Original Message- From: Tim Donohue [mailto:tdono...@duraspace.org] Sent: Tuesday, April 07, 2015 2:33 PM To: George Stanley Kozak; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, On 4/7/2015 1:09 PM, George Stanley Kozak wrote: Hi, Tim: 1) I did the lucene-core-3.5.0.jar manual update of the indexes. For the search index, I got no errors, but for the statistics core, I got: Exception in thread main org.apache.lucene.index.IndexNotFoundException: org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65 at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118) at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85) This would imply that the search index upgraded fine (no errors). But, the error for your statistics index almost implies that it's not a valid Solr index. Do you use the statistics index? Is there anything in the /cul/app/dspace/solr/statistics/data/index directory or is it empty? 2) I deleted the statistics core and did the ant update. I brought up my DSpace 5.1 test system and when I tried to do a search, I got no results. When I went to do a browse, I got: ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ anonymous:session_id=5749DBA88953D8B0ABDDA3FFD1DFF5F2:ip_addr=128.84.1 17.195:Error in Discovery while setting up date facet range:date facet\colon; dateIssued.year SolrCore 'search' is not available due to init failure: Index locked for write for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search' is not available due to init failure: Index locked for write for core search 3) I stopped tomcat, deleted the search index (as you suggested) and did the mvn build and ant update and then copied the new webapps over to tomcat. Everything looked good. I started tomcat and my site came up. Before doing anything more I ran ./dspace index-discovery -b. It ran to
Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information)
Hey there, Sorry for not being descriptive enough. In tomcat.xml, there's a Host section where the DSpace app context fragments go: ___ Host name=localhost appBase=/opt/dspace/webapps unpackWARs=true autoDeploy=true xmlValidation=false xmlNamespaceAware=false !-- DEFINE A CONTEXT PATH FOR DSpace XML User Interface (Manakin) -- Context path=/ docBase=/opt/dspace/webapps/xmlui debug=0 reloadable=true cachingAllowed=false allowLinking=true/ ___ In that first line, the appBase may still be pointed to Tomcat's default. For us, it let DSpace start and function semi-well, but none of the SOLR stuff would work. B-- -Original Message- From: George Stanley Kozak [mailto:g...@cornell.edu] Sent: Tuesday, April 7, 2015 2:04 PM To: Brian Freels-Stendel; dspace-tech@lists.sourceforge.net Subject: RE: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi, Brian: Sorry for being dense, but what do you mean by updating the appBase directory? George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -Original Message- From: Brian Freels-Stendel [mailto:bfre...@unm.edu] Sent: Tuesday, April 07, 2015 3:47 PM To: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Good afternoon, We saw very similar errors (particularly Expected mime type application/octet-stream but got text/html) and found that we hadn't updated the appBase directory in tomcat.xml. Another symptom of ours was the code of the entire error page being written to the dspace log. It's a longshot, but perhaps something to check. B-- -Original Message- From: George Stanley Kozak [mailto:g...@cornell.edu] Sent: Tuesday, April 7, 2015 1:28 PM To: Tim Donohue; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi, Tim: To answer one of your questions: my original DSpace 1.8.2 solr statistics core did have data in it. Attached is a snapshot of the index directory. Now, I did as you asked. I deleted the all of the solr cores (actually everything under /cul/app/dspace/solr) I did the mvn and ant update and everything went well. I copied the webapps over to tomcat and restarted tomcat. I watched the DSpace log and the catalina.out log. Everything seemed to move along smoothly. When everything was done, I waited a few minutes before I went to my DSpace home page. When my home page came up, I began to see errors in my DSpace log: ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ anonymous:session_id=B3B12E105B95C8FB8E63E9FB94B5971F:ip_addr=128.84.117.195:Error in Discovery while setting up date facet range:date facet\colon; dateIssued.year SolrCore 'search' is not available due to init failure: Index locked for write for core search,trace=org.apache.solr.common.SolrException: SolrCore 'search' is not available due to init failure: Index locked for write for core search Any attempt to do any browse or search results with errors. By the way, the errors on the web appear with org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. And then raw HTML after that. What has me stumped is that outside of the problem with the solr statistics, none of these problems appear in my DSpace 4.3 upgrade. I have no write lock problems in DSpace 4.3 install. George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 218 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -Original Message- From: Tim Donohue [mailto:tdono...@duraspace.org] Sent: Tuesday, April 07, 2015 2:33 PM To: George Stanley Kozak; dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error after install of DSpace 5.1 (Additional information) Hi George, On 4/7/2015 1:09 PM, George Stanley Kozak wrote: Hi, Tim: 1) I did the lucene-core-3.5.0.jar manual update of the indexes. For the search index, I got no errors, but for the statistics core, I got: Exception in thread main org.apache.lucene.index.IndexNotFoundException: org.apache.lucene.store.MMapDirectory@/cul/app/dspace/solr/statistics/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e888d65 at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:118) at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:85) This would imply that the search index upgraded fine (no errors). But, the error for your statistics index almost implies that it's not a valid Solr index. Do you use the statistics index? Is there anything in the /cul/app/dspace/solr/statistics/data/index directory
