Dear FILIPPOS,
Thanks a lot for your advice. I have tried both of your suggested
approaches:
1. I ran the query and found that there is no record with "*withdrawn*"
marked as "t". All are "f".
2. Running the wget command returns 0 item without handle number:
{
"responseHeader":{
"status":0,
"QTime":1,
"params":{
"q":"*:* AND -handle:{* TO *}",
"indent":"true",
"wt":"json"}},
"response":{"numFound":0,"start":0,"docs":[]
}}
I also recreated the SOLR index with* /dspace/bin/dspace index-discovery
-b -f* followed by */dspace/bin/dspace oai import -c*. And I still got the
same error.
Anyway, your kind advice is much appreciated.
Panyarak
On Tue, Mar 23, 2021 at 6:58 PM FILIPPOS KOLOVOS <[email protected]> wrote:
> Dear Sir,
>
> I am glad that I have helped you.
>
> This error with the XOAI import happened also to our installation a few
> years ago and sometimes it happens because it tries to index in OAI an item
> from the SOLR index that *does not have a handle number*. This happens
> when an item with no handle number has been indexed in SOLR, which should
> not occur.
>
> This might not happen for an ordinary item, but it may occur for an* Item
> Template* that is specified for a collection, but it is *withdrawn *(which
> should *NOT *happen).
> You should check all your *withdrawn *items in DSpace and if you find one
> or more that are not ordinary items, but* Item Templates*, then you
> should delete and recreate them for the collection(s) they belong to. *Make
> sure to keep the pre-filled data, so that you will be able to re-create
> them.*
>
> One query you could run in your postgres, in order to see which *items in
> general *do not have a handle number is:
> *SELECT * FROM item WHERE NOT EXISTS (SELECT resource_id FROM handle WHERE
> handle.resource_id = item.uuid AND handle.resource_type_id = 2);*
>
> The query will respond with a number of items with *no handle number*.
> For most of the items is *normal*, since they might be incomplete
> submissions from users, or submissions which have not completed the
> workflow, etc. You will also see item templates in that list. They also do
> not have handle numbers and it is normal.
> However, check the "*withdrawn*" column for everyone of these items and
> to see if it is "t" (true) and for these items check if they are *item
> templates*. You can do that by pasting the uuid in the "*internal ID*" of
> the administrative interface of DSpace in "*Content->Items*" If that is
> the case, then you must delete them and recreate them for the collection
> they belong to, but this time they must not be withdrawn. When you recreate
> them, before you run the OAI index script, you will have to recreate the
> SOLR index with
> */dspace/bin/index-discovery -b -f *so that the erroneous item will be
> excluded from the index (the* -b deletes the index *and the *-f forces
> each item to be re-indexed*, so make sure that your reindexing process
> works well, or have a backup of the index before attempting this necessary
> step, or you might end up with no search index if it also crashes).
>
> If that does not solve the problem, then another item with no handle
> number might be the culprit, ordinary or Item template, which mistakenly
> has been indexed in SOLR.
>
> In case you would like to dive into SOLR, there is this query that can be
> run *directly to SOLR from the console of your Linux server*, which will
> return* all indexed documents in SOLR that do not have a handle number.*
> However, the returned file is an XML JSON file, but I guess it will provide
> enough information about these erroneous items and then you can spot them.
> The query is (keep the single and double quotes, or the shell might not ran
> the command).:
>
> *wget 'http://127.0.0.1:8080/solr/search/select?q=*:*+AND+-handle:{*
> <http://127.0.0.1:8080/solr/search/select?q=*:*+AND+-handle:%7B*> TO
> *}&wt=json&indent=true'*
>
> I hope this also helps you
>
> Best Regards,
>
> -Fk
>
> On Tue, Mar 23, 2021 at 11:56 AM Panyarak Ngamsritragul <
> [email protected]> wrote:
>
>>
>> Thanks a lot Filippos. That really help solve my problem!
>>
>> Anyway, I got another error following the success of importing the items:
>> 11400 items imported so far...
>> 11500 items imported so far...
>> Total: 11505 items
>> java.lang.NullPointerException
>> at org.dspace.xoai.app.XOAI.willChangeStatus(XOAI.java:438)
>> at org.dspace.xoai.app.XOAI.index(XOAI.java:368)
>> at org.dspace.xoai.app.XOAI.index(XOAI.java:280)
>> at org.dspace.xoai.app.XOAI.indexAll(XOAI.java:227)
>> at org.dspace.xoai.app.XOAI.index(XOAI.java:134)
>> at org.dspace.xoai.app.XOAI.main(XOAI.java:560)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
>> at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
>>
>> I did searches through the web and found that it will be resolved in
>> 6.4...
>> https://alanorth.github.io/cgspace-notes/2020-06/
>>
>> Thanks a lot.
>> Panyarak
>>
>> On Mon, Mar 22, 2021 at 11:16 PM FILIPPOS KOLOVOS <[email protected]>
>> wrote:
>>
>>> Dear Sir,
>>>
>>> This error relates to the Garbage Collection (GC) mechanism of JVM,
>>> which means that it does not have enough memory available to complete the
>>> task. More in particular, it means that the Garbage Collector is spending
>>> too much time clearing the heap space from unused objects, but at the same
>>> time it does not free more than 2% of the heap space, which is used for the
>>> instantiated objects. That's wh the GC reports that it does not have any
>>> more space to do its job.
>>>
>>> One thing you can do, if you do not have enough RAM on your server, is
>>> first shutdown Tomcat (this will lead to some downtime for your server, but
>>> it will free up some valuable RAM and after OAI completes you can restart
>>> it) and then go to the */dspace/bin/dspace* file of your running dspace
>>> instance and edit it with your text editor. Then, in the file search and
>>> find the line:
>>>
>>> *JAVA_OPTS="-Xmx256m -Dfile.encoding=UTF-8"*
>>>
>>> and *change it *to:
>>>
>>> *JAVA_OPTS="-Xmx2048m -Xms1024m -Dfile.encoding=UTF-8"*
>>>
>>> *DO NOT* copy and paste the line, but edit it in place, because you do
>>> not want to mess up this file.
>>>
>>> This will specify that the dspace *TOOL *(which is the file you are
>>> running for the OAI) will use *from 1024MB up to 2048MB RAM* for its
>>> task and not just *256MB*. However, you *must have more than 2048MB RAM*
>>> available on the server (i.e. 3GB), or the script will fail again and it
>>> might hang your server. If you have *LESS *than 3GB allocated to the
>>> server, then adjust these values accordingly (i.e. Xmx1024m Xms500m, etc).
>>>
>>> The bottomline is that you do not have enough memory to complete the
>>> task and even if your server has 32GB RAM, the dspace tool will only use up
>>> to the RAM specified in this file.
>>>
>>> I hope that I have helped!
>>>
>>> Best Regards,
>>>
>>> -Fk
>>>
>>> On Mon, Mar 22, 2021 at 4:08 PM Panyarak Ngamsritragul <
>>> [email protected]> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I am using DSpace 6.3+Apache Tomcat Version 8.0.37+javac 11.0.10
>>>>
>>>> The instance I am managing has 11,428 records. (kb.psu.ac.th)
>>>>
>>>> I tried to create indexes for OAI using this command
>>>>
>>>> /dspace/bin/dspace oai import -c
>>>>
>>>> It worked until 8900 items and crashed with error messages:
>>>>
>>>> 8600 items imported so far...
>>>> 8700 items imported so far...
>>>> 8800 items imported so far...
>>>> 8900 items imported so far...
>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>> at java.util.Arrays.copyOfRange(Arrays.java:3664)
>>>> at java.lang.String.<init>(String.java:207)
>>>> at java.lang.StringBuilder.toString(StringBuilder.java:407)
>>>> at
>>>> org.hibernate.persister.entity.AbstractEntityPersister.selectFragment(AbstractEntityPersister.java:1422)
>>>> at
>>>> org.hibernate.persister.entity.AbstractEntityPersister.selectFragment(AbstractEntityPersister.java:4434)
>>>> at
>>>> org.hibernate.loader.JoinWalker.selectString(JoinWalker.java:1099)
>>>> at
>>>> org.hibernate.loader.AbstractEntityJoinWalker.initStatementString(AbstractEntityJoinWalker.java:123)
>>>> at
>>>> org.hibernate.loader.AbstractEntityJoinWalker.initStatementString(AbstractEntityJoinWalker.java:108)
>>>> at
>>>> org.hibernate.loader.AbstractEntityJoinWalker.initAll(AbstractEntityJoinWalker.java:90)
>>>> at
>>>> org.hibernate.loader.AbstractEntityJoinWalker.initAll(AbstractEntityJoinWalker.java:77)
>>>> at
>>>> org.hibernate.loader.criteria.CriteriaJoinWalker.<init>(CriteriaJoinWalker.java:123)
>>>> at
>>>> org.hibernate.loader.criteria.CriteriaJoinWalker.<init>(CriteriaJoinWalker.java:92)
>>>> at
>>>> org.hibernate.loader.criteria.CriteriaLoader.<init>(CriteriaLoader.java:95)
>>>> at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1604)
>>>> at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:374)
>>>> at
>>>> org.dspace.core.AbstractHibernateDAO.list(AbstractHibernateDAO.java:158)
>>>> at
>>>> org.dspace.authorize.dao.impl.ResourcePolicyDAOImpl.findByDSoAndAction(ResourcePolicyDAOImpl.java:74)
>>>> at
>>>> org.dspace.authorize.ResourcePolicyServiceImpl.find(ResourcePolicyServiceImpl.java:103)
>>>> at
>>>> org.dspace.authorize.AuthorizeServiceImpl.getPoliciesActionFilter(AuthorizeServiceImpl.java:575)
>>>> at
>>>> org.dspace.authorize.AuthorizeServiceImpl.authorize(AuthorizeServiceImpl.java:301)
>>>> at
>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeAction(AuthorizeServiceImpl.java:129)
>>>> at
>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeAction(AuthorizeServiceImpl.java:95)
>>>> at
>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeActionBoolean(AuthorizeServiceImpl.java:181)
>>>> at
>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeActionBoolean(AuthorizeServiceImpl.java:166)
>>>> at org.dspace.xoai.app.XOAI.isPublic(XOAI.java:458)
>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:343)
>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:280)
>>>> at org.dspace.xoai.app.XOAI.indexAll(XOAI.java:227)
>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:134)
>>>> at org.dspace.xoai.app.XOAI.main(XOAI.java:560)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>
>>>> Could you please help to solve this problem?
>>>>
>>>> Thanks and best regards,
>>>> Panyarak Ngamsritragul
>>>>
>>>> --
>>>> All messages to this mailing list should adhere to the Code of Conduct:
>>>> https://duraspace.org/about/policies/code-of-conduct/
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "DSpace Community" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/dspace-community/fcc5dfb5-0c01-4665-b490-e915d4662f96n%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/dspace-community/fcc5dfb5-0c01-4665-b490-e915d4662f96n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>
>>>
--
All messages to this mailing list should adhere to the Code of Conduct:
https://duraspace.org/about/policies/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups
"DSpace Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/dspace-community/CAJc8HTeAbFCLabLuh%3DOU15OiDbGUmjsb3_dwGbwiFKXBrTmT%3Dw%40mail.gmail.com.
