Dear FILIPPOS, Thanks again for your advice. I tried your suggestion right away, but still no luck. The same error still persists.
BTW, your suggestions help me learn a lot more. Thank you. Panyarak On Thu, Mar 25, 2021 at 1:48 AM FILIPPOS KOLOVOS <[email protected]> wrote: > Dear Sir, > > I took a second look in your previous reply, where you provide a link to > this bug which is fixed in DSpace 6.4. I think I should have taken a look > earlier, because it analyzes the erroneous code. > > I also run two DSpace installations, one of which is version 6.3. I > frequently work on the source code, in order to apply customizations and / > or bug fixes. So, I applied the patch for XOAI in the 6.3 DSpace > installation and I sent you the *compiled XOAI.class* file, in order for > you to try it out and see if it works. > > My Installation is using *Tomcat 9.0.24 and JAVA 8u191*. > > 1. You will have to first *shutdown Tomcat* > 2. Then, you must *replace *the file > > *[running-dspace-directory]/webapps/oai/WEB-INF/classes/org/dspace/xoai/app/XOAI.class* > with the attached one and it should work fine (the > *running-dspace-directory > *is the root directory of your Dspace running instance). However,* > just in case the java versions do not match* (which I do not think > so), keep a *backup *of the *old XOAI.class file*. > 3. Then check in your Tomcat directory, if this class is in the > Tomcat's *running cache* and delete it, in order for the new file to > take effect. This cache is normally at > */usr/local/apache-tomcat-version/work/Catalina/localhost/oai/XOAI.class. > *Same here, the *apache-tomcat-version* is the directory of your > version of Tomcat. > 4. Restart Tomcat > > > This time I hope that you will solve your problem and finally reindex OAI. > > Best Regards, > > -Fk > > > On Wed, Mar 24, 2021 at 11:50 AM Panyarak Ngamsritragul < > [email protected]> wrote: > >> Dear FILIPPOS, >> >> Thanks a lot for your advice. I have tried both of your suggested >> approaches: >> 1. I ran the query and found that there is no record with "*withdrawn*" >> marked as "t". All are "f". >> 2. Running the wget command returns 0 item without handle number: >> { >> "responseHeader":{ >> "status":0, >> "QTime":1, >> "params":{ >> "q":"*:* AND -handle:{* TO *}", >> "indent":"true", >> "wt":"json"}}, >> "response":{"numFound":0,"start":0,"docs":[] >> }} >> >> I also recreated the SOLR index with* /dspace/bin/dspace >> index-discovery -b -f* followed by */dspace/bin/dspace oai import -c*. >> And I still got the same error. >> >> Anyway, your kind advice is much appreciated. >> >> Panyarak >> >> On Tue, Mar 23, 2021 at 6:58 PM FILIPPOS KOLOVOS <[email protected]> wrote: >> >>> Dear Sir, >>> >>> I am glad that I have helped you. >>> >>> This error with the XOAI import happened also to our installation a few >>> years ago and sometimes it happens because it tries to index in OAI an item >>> from the SOLR index that *does not have a handle number*. This happens >>> when an item with no handle number has been indexed in SOLR, which should >>> not occur. >>> >>> This might not happen for an ordinary item, but it may occur for an* >>> Item Template* that is specified for a collection, but it is *withdrawn >>> *(which should *NOT *happen). >>> You should check all your *withdrawn *items in DSpace and if you find >>> one or more that are not ordinary items, but* Item Templates*, then you >>> should delete and recreate them for the collection(s) they belong to. *Make >>> sure to keep the pre-filled data, so that you will be able to re-create >>> them.* >>> >>> One query you could run in your postgres, in order to see which *items >>> in general *do not have a handle number is: >>> *SELECT * FROM item WHERE NOT EXISTS (SELECT resource_id FROM handle >>> WHERE handle.resource_id = item.uuid AND handle.resource_type_id = 2);* >>> >>> The query will respond with a number of items with *no handle number*. >>> For most of the items is *normal*, since they might be incomplete >>> submissions from users, or submissions which have not completed the >>> workflow, etc. You will also see item templates in that list. They also do >>> not have handle numbers and it is normal. >>> However, check the "*withdrawn*" column for everyone of these items and >>> to see if it is "t" (true) and for these items check if they are *item >>> templates*. You can do that by pasting the uuid in the "*internal ID*" >>> of the administrative interface of DSpace in "*Content->Items*" If that >>> is the case, then you must delete them and recreate them for the collection >>> they belong to, but this time they must not be withdrawn. When you recreate >>> them, before you run the OAI index script, you will have to recreate the >>> SOLR index with >>> */dspace/bin/index-discovery -b -f *so that the erroneous item will be >>> excluded from the index (the* -b deletes the index *and the *-f forces >>> each item to be re-indexed*, so make sure that your reindexing process >>> works well, or have a backup of the index before attempting this necessary >>> step, or you might end up with no search index if it also crashes). >>> >>> If that does not solve the problem, then another item with no handle >>> number might be the culprit, ordinary or Item template, which mistakenly >>> has been indexed in SOLR. >>> >>> In case you would like to dive into SOLR, there is this query that can >>> be run *directly to SOLR from the console of your Linux server*, which >>> will return* all indexed documents in SOLR that do not have a handle >>> number.* However, the returned file is an XML JSON file, but I guess it >>> will provide enough information about these erroneous items and then you >>> can spot them. The query is (keep the single and double quotes, or the >>> shell might not ran the command).: >>> >>> *wget 'http://127.0.0.1:8080/solr/search/select?q=*:*+AND+-handle:{* >>> <http://127.0.0.1:8080/solr/search/select?q=*:*+AND+-handle:%7B*> TO >>> *}&wt=json&indent=true'* >>> >>> I hope this also helps you >>> >>> Best Regards, >>> >>> -Fk >>> >>> On Tue, Mar 23, 2021 at 11:56 AM Panyarak Ngamsritragul < >>> [email protected]> wrote: >>> >>>> >>>> Thanks a lot Filippos. That really help solve my problem! >>>> >>>> Anyway, I got another error following the success of importing the >>>> items: >>>> 11400 items imported so far... >>>> 11500 items imported so far... >>>> Total: 11505 items >>>> java.lang.NullPointerException >>>> at org.dspace.xoai.app.XOAI.willChangeStatus(XOAI.java:438) >>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:368) >>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:280) >>>> at org.dspace.xoai.app.XOAI.indexAll(XOAI.java:227) >>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:134) >>>> at org.dspace.xoai.app.XOAI.main(XOAI.java:560) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>> at >>>> org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229) >>>> at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81) >>>> >>>> I did searches through the web and found that it will be resolved in >>>> 6.4... >>>> https://alanorth.github.io/cgspace-notes/2020-06/ >>>> >>>> Thanks a lot. >>>> Panyarak >>>> >>>> On Mon, Mar 22, 2021 at 11:16 PM FILIPPOS KOLOVOS <[email protected]> >>>> wrote: >>>> >>>>> Dear Sir, >>>>> >>>>> This error relates to the Garbage Collection (GC) mechanism of JVM, >>>>> which means that it does not have enough memory available to complete the >>>>> task. More in particular, it means that the Garbage Collector is spending >>>>> too much time clearing the heap space from unused objects, but at the same >>>>> time it does not free more than 2% of the heap space, which is used for >>>>> the >>>>> instantiated objects. That's wh the GC reports that it does not have any >>>>> more space to do its job. >>>>> >>>>> One thing you can do, if you do not have enough RAM on your server, is >>>>> first shutdown Tomcat (this will lead to some downtime for your server, >>>>> but >>>>> it will free up some valuable RAM and after OAI completes you can restart >>>>> it) and then go to the */dspace/bin/dspace* file of your running >>>>> dspace instance and edit it with your text editor. Then, in the file >>>>> search >>>>> and find the line: >>>>> >>>>> *JAVA_OPTS="-Xmx256m -Dfile.encoding=UTF-8"* >>>>> >>>>> and *change it *to: >>>>> >>>>> *JAVA_OPTS="-Xmx2048m -Xms1024m -Dfile.encoding=UTF-8"* >>>>> >>>>> *DO NOT* copy and paste the line, but edit it in place, because you >>>>> do not want to mess up this file. >>>>> >>>>> This will specify that the dspace *TOOL *(which is the file you are >>>>> running for the OAI) will use *from 1024MB up to 2048MB RAM* for its >>>>> task and not just *256MB*. However, you *must have more than 2048MB >>>>> RAM* available on the server (i.e. 3GB), or the script will fail >>>>> again and it might hang your server. If you have *LESS *than 3GB >>>>> allocated to the server, then adjust these values accordingly (i.e. >>>>> Xmx1024m Xms500m, etc). >>>>> >>>>> The bottomline is that you do not have enough memory to complete the >>>>> task and even if your server has 32GB RAM, the dspace tool will only use >>>>> up >>>>> to the RAM specified in this file. >>>>> >>>>> I hope that I have helped! >>>>> >>>>> Best Regards, >>>>> >>>>> -Fk >>>>> >>>>> On Mon, Mar 22, 2021 at 4:08 PM Panyarak Ngamsritragul < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am using DSpace 6.3+Apache Tomcat Version 8.0.37+javac 11.0.10 >>>>>> >>>>>> The instance I am managing has 11,428 records. (kb.psu.ac.th) >>>>>> >>>>>> I tried to create indexes for OAI using this command >>>>>> >>>>>> /dspace/bin/dspace oai import -c >>>>>> >>>>>> It worked until 8900 items and crashed with error messages: >>>>>> >>>>>> 8600 items imported so far... >>>>>> 8700 items imported so far... >>>>>> 8800 items imported so far... >>>>>> 8900 items imported so far... >>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>>> at java.util.Arrays.copyOfRange(Arrays.java:3664) >>>>>> at java.lang.String.<init>(String.java:207) >>>>>> at java.lang.StringBuilder.toString(StringBuilder.java:407) >>>>>> at >>>>>> org.hibernate.persister.entity.AbstractEntityPersister.selectFragment(AbstractEntityPersister.java:1422) >>>>>> at >>>>>> org.hibernate.persister.entity.AbstractEntityPersister.selectFragment(AbstractEntityPersister.java:4434) >>>>>> at >>>>>> org.hibernate.loader.JoinWalker.selectString(JoinWalker.java:1099) >>>>>> at >>>>>> org.hibernate.loader.AbstractEntityJoinWalker.initStatementString(AbstractEntityJoinWalker.java:123) >>>>>> at >>>>>> org.hibernate.loader.AbstractEntityJoinWalker.initStatementString(AbstractEntityJoinWalker.java:108) >>>>>> at >>>>>> org.hibernate.loader.AbstractEntityJoinWalker.initAll(AbstractEntityJoinWalker.java:90) >>>>>> at >>>>>> org.hibernate.loader.AbstractEntityJoinWalker.initAll(AbstractEntityJoinWalker.java:77) >>>>>> at >>>>>> org.hibernate.loader.criteria.CriteriaJoinWalker.<init>(CriteriaJoinWalker.java:123) >>>>>> at >>>>>> org.hibernate.loader.criteria.CriteriaJoinWalker.<init>(CriteriaJoinWalker.java:92) >>>>>> at >>>>>> org.hibernate.loader.criteria.CriteriaLoader.<init>(CriteriaLoader.java:95) >>>>>> at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1604) >>>>>> at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:374) >>>>>> at >>>>>> org.dspace.core.AbstractHibernateDAO.list(AbstractHibernateDAO.java:158) >>>>>> at >>>>>> org.dspace.authorize.dao.impl.ResourcePolicyDAOImpl.findByDSoAndAction(ResourcePolicyDAOImpl.java:74) >>>>>> at >>>>>> org.dspace.authorize.ResourcePolicyServiceImpl.find(ResourcePolicyServiceImpl.java:103) >>>>>> at >>>>>> org.dspace.authorize.AuthorizeServiceImpl.getPoliciesActionFilter(AuthorizeServiceImpl.java:575) >>>>>> at >>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorize(AuthorizeServiceImpl.java:301) >>>>>> at >>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeAction(AuthorizeServiceImpl.java:129) >>>>>> at >>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeAction(AuthorizeServiceImpl.java:95) >>>>>> at >>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeActionBoolean(AuthorizeServiceImpl.java:181) >>>>>> at >>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeActionBoolean(AuthorizeServiceImpl.java:166) >>>>>> at org.dspace.xoai.app.XOAI.isPublic(XOAI.java:458) >>>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:343) >>>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:280) >>>>>> at org.dspace.xoai.app.XOAI.indexAll(XOAI.java:227) >>>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:134) >>>>>> at org.dspace.xoai.app.XOAI.main(XOAI.java:560) >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>> at >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>> >>>>>> Could you please help to solve this problem? >>>>>> >>>>>> Thanks and best regards, >>>>>> Panyarak Ngamsritragul >>>>>> >>>>>> -- >>>>>> All messages to this mailing list should adhere to the Code of >>>>>> Conduct: https://duraspace.org/about/policies/code-of-conduct/ >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "DSpace Community" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/dspace-community/fcc5dfb5-0c01-4665-b490-e915d4662f96n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/dspace-community/fcc5dfb5-0c01-4665-b490-e915d4662f96n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>> >>>>> -- All messages to this mailing list should adhere to the Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/ --- You received this message because you are subscribed to the Google Groups "DSpace Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-community/CAJc8HTdsMjcFjSUexF5od_NB3F9bY4N8XpohWDjPmPBTf%3D_x7A%40mail.gmail.com.
