Dear FILIPPOS I have tried your new file, but still got the same error. Well, what I am suspecting is whether I did it correctly.
The Tomcat I am using was manually installed. So the file XOAI.class should be placed in /opt/tomcat/webapps/oai/WEB-INF/classes/org/dspace/xoai/app and I found no cache of XOAI.class in /opt/tomcat/work/Catalina/localhost/oai. I wonder if Tomcat place the cache files in some other locations... This is what I got when I list the content of directories: panya@kb:/opt/tomcat/work/Catalina/localhost/oai$ ls -la total 8 drwxr-x--- 2 tomcat tomcat 4096 Jun 24 2020 . drwxr-x--- 16 tomcat tomcat 4096 Nov 5 08:27 .. panya@kb:/opt/tomcat/work/Catalina/localhost/oai$ panya@kb1:/opt/tomcat/webapps/oai/WEB-INF/classes/org/dspace/xoai/app$ ls -la total 44 drwxr-xr-x 2 tomcat tomcat 4096 ต.ค. 1 11:39 . drwxr-xr-x 10 tomcat tomcat 4096 ต.ค. 1 11:39 .. -rw-r--r-- 1 tomcat tomcat 5364 ต.ค. 1 11:39 BasicConfiguration.class -rw-r--r-- 1 tomcat tomcat 2881 ต.ค. 1 11:39 DSpaceWebappConfiguration.class -rw-r--r-- 1 tomcat tomcat 24350 มี.ค. 26 15:12 XOAI.class panya@kb1:/opt/tomcat/webapps/oai/WEB-INF/classes/org/dspace/xoai/app$ Maybe you can modify the first output line of XOAI.class so that I can check which version of XOAI.class Tomcat is calling... OAI 2.0 manager action started <--- Can you modify this? Clearing index Index cleared Using full import. Full import 100 items imported so far... 200 items imported so far... Thanks a lot. Panyarak On Thu, Mar 25, 2021 at 7:38 PM FILIPPOS KOLOVOS <[email protected]> wrote: > Hm, that's odd. Are you sure that the *old XOAI.class* is not still > running in Tomcat's cache? Did you check for it, deleted it and did a > shutdown-start cycle during the XOAI update? The *new XOAI.class* file > should be *24.328* bytes and the old one 24.312. > > Because the error that it throws is *exactly *at line 438 in the method > *willChangeStatus() *and it is a *nullPointerException. Exactly *at that > line (and at 331 but that's not where it throws the error in your case) I > have inserted the patched code that *explicitly* checks if the policy's > group is null *before *doing anything else. It shouldn't throw that error > that I include below. > > The only other reason that I can think of that leads to the same error, is > that you have item policies with valid groups (not null), but with *null > names* and so it crashes at the next step where it checks for > *policy.getGroup().getName()*. I "enhanced" the *XOAI.class patch* a bit, > by inserting an *additional *check for the *policy's name as well, before* > checking it's name (*policy.getGroup().getName().equals("Anonymous")*), > because that is the only other place that it could throw an exception like > this. > > I recompiled it and I am attaching it again for you to try it out if you > wish. The new XOAI.class file should be *24.350* bytes. For such a small > patch, its a pitty not to be able to solve that error. > > Error Throwed: > > > > > > > > > > > > > > > *Total: 11505 itemsjava.lang.NullPointerExceptionat > org.dspace.xoai.app.XOAI.willChangeStatus(XOAI.java:438)at > org.dspace.xoai.app.XOAI.index(XOAI.java:368)at > org.dspace.xoai.app.XOAI.index(XOAI.java:280)at > org.dspace.xoai.app.XOAI.indexAll(XOAI.java:227)at > org.dspace.xoai.app.XOAI.index(XOAI.java:134)at > org.dspace.xoai.app.XOAI.main(XOAI.java:560)at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at > java.lang.reflect.Method.invoke(Method.java:498)at > org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)* > *at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)* > > On Thu, Mar 25, 2021 at 9:01 AM Panyarak Ngamsritragul <[email protected]> > wrote: > >> Dear FILIPPOS, >> >> Thanks again for your advice. >> I tried your suggestion right away, but still no luck. The same error >> still persists. >> >> BTW, your suggestions help me learn a lot more. Thank you. >> >> Panyarak >> >> On Thu, Mar 25, 2021 at 1:48 AM FILIPPOS KOLOVOS <[email protected]> wrote: >> >>> Dear Sir, >>> >>> I took a second look in your previous reply, where you provide a link to >>> this bug which is fixed in DSpace 6.4. I think I should have taken a look >>> earlier, because it analyzes the erroneous code. >>> >>> I also run two DSpace installations, one of which is version 6.3. I >>> frequently work on the source code, in order to apply customizations and / >>> or bug fixes. So, I applied the patch for XOAI in the 6.3 DSpace >>> installation and I sent you the *compiled XOAI.class* file, in order >>> for you to try it out and see if it works. >>> >>> My Installation is using *Tomcat 9.0.24 and JAVA 8u191*. >>> >>> 1. You will have to first *shutdown Tomcat* >>> 2. Then, you must *replace *the file >>> >>> *[running-dspace-directory]/webapps/oai/WEB-INF/classes/org/dspace/xoai/app/XOAI.class* >>> with the attached one and it should work fine (the >>> *running-dspace-directory >>> *is the root directory of your Dspace running instance). However,* >>> just in case the java versions do not match* (which I do not think >>> so), keep a *backup *of the *old XOAI.class file*. >>> 3. Then check in your Tomcat directory, if this class is in the >>> Tomcat's *running cache* and delete it, in order for the new file to >>> take effect. This cache is normally at >>> */usr/local/apache-tomcat-version/work/Catalina/localhost/oai/XOAI.class. >>> *Same here, the *apache-tomcat-version* is the directory of your >>> version of Tomcat. >>> 4. Restart Tomcat >>> >>> >>> This time I hope that you will solve your problem and finally reindex >>> OAI. >>> >>> Best Regards, >>> >>> -Fk >>> >>> >>> On Wed, Mar 24, 2021 at 11:50 AM Panyarak Ngamsritragul < >>> [email protected]> wrote: >>> >>>> Dear FILIPPOS, >>>> >>>> Thanks a lot for your advice. I have tried both of your suggested >>>> approaches: >>>> 1. I ran the query and found that there is no record with "*withdrawn*" >>>> marked as "t". All are "f". >>>> 2. Running the wget command returns 0 item without handle number: >>>> { >>>> "responseHeader":{ >>>> "status":0, >>>> "QTime":1, >>>> "params":{ >>>> "q":"*:* AND -handle:{* TO *}", >>>> "indent":"true", >>>> "wt":"json"}}, >>>> "response":{"numFound":0,"start":0,"docs":[] >>>> }} >>>> >>>> I also recreated the SOLR index with* /dspace/bin/dspace >>>> index-discovery -b -f* followed by */dspace/bin/dspace oai import -c*. >>>> And I still got the same error. >>>> >>>> Anyway, your kind advice is much appreciated. >>>> >>>> Panyarak >>>> >>>> On Tue, Mar 23, 2021 at 6:58 PM FILIPPOS KOLOVOS <[email protected]> >>>> wrote: >>>> >>>>> Dear Sir, >>>>> >>>>> I am glad that I have helped you. >>>>> >>>>> This error with the XOAI import happened also to our installation a >>>>> few years ago and sometimes it happens because it tries to index in OAI an >>>>> item from the SOLR index that *does not have a handle number*. This >>>>> happens when an item with no handle number has been indexed in SOLR, which >>>>> should not occur. >>>>> >>>>> This might not happen for an ordinary item, but it may occur for an* >>>>> Item Template* that is specified for a collection, but it is *withdrawn >>>>> *(which should *NOT *happen). >>>>> You should check all your *withdrawn *items in DSpace and if you find >>>>> one or more that are not ordinary items, but* Item Templates*, then >>>>> you should delete and recreate them for the collection(s) they belong to. >>>>> *Make >>>>> sure to keep the pre-filled data, so that you will be able to re-create >>>>> them.* >>>>> >>>>> One query you could run in your postgres, in order to see which *items >>>>> in general *do not have a handle number is: >>>>> *SELECT * FROM item WHERE NOT EXISTS (SELECT resource_id FROM handle >>>>> WHERE handle.resource_id = item.uuid AND handle.resource_type_id = 2);* >>>>> >>>>> The query will respond with a number of items with *no handle number*. >>>>> For most of the items is *normal*, since they might be incomplete >>>>> submissions from users, or submissions which have not completed the >>>>> workflow, etc. You will also see item templates in that list. They also do >>>>> not have handle numbers and it is normal. >>>>> However, check the "*withdrawn*" column for everyone of these items >>>>> and to see if it is "t" (true) and for these items check if they are *item >>>>> templates*. You can do that by pasting the uuid in the "*internal ID*" >>>>> of the administrative interface of DSpace in "*Content->Items*" If >>>>> that is the case, then you must delete them and recreate them for the >>>>> collection they belong to, but this time they must not be withdrawn. When >>>>> you recreate them, before you run the OAI index script, you will have to >>>>> recreate the SOLR index with >>>>> */dspace/bin/index-discovery -b -f *so that the erroneous item will >>>>> be excluded from the index (the* -b deletes the index *and the *-f >>>>> forces each item to be re-indexed*, so make sure that your reindexing >>>>> process works well, or have a backup of the index before attempting this >>>>> necessary step, or you might end up with no search index if it also >>>>> crashes). >>>>> >>>>> If that does not solve the problem, then another item with no handle >>>>> number might be the culprit, ordinary or Item template, which mistakenly >>>>> has been indexed in SOLR. >>>>> >>>>> In case you would like to dive into SOLR, there is this query that can >>>>> be run *directly to SOLR from the console of your Linux server*, >>>>> which will return* all indexed documents in SOLR that do not have a >>>>> handle number.* However, the returned file is an XML JSON file, but I >>>>> guess it will provide enough information about these erroneous items and >>>>> then you can spot them. The query is (keep the single and double quotes, >>>>> or >>>>> the shell might not ran the command).: >>>>> >>>>> *wget 'http://127.0.0.1:8080/solr/search/select?q=*:*+AND+-handle:{* >>>>> <http://127.0.0.1:8080/solr/search/select?q=*:*+AND+-handle:%7B*> TO >>>>> *}&wt=json&indent=true'* >>>>> >>>>> I hope this also helps you >>>>> >>>>> Best Regards, >>>>> >>>>> -Fk >>>>> >>>>> On Tue, Mar 23, 2021 at 11:56 AM Panyarak Ngamsritragul < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> Thanks a lot Filippos. That really help solve my problem! >>>>>> >>>>>> Anyway, I got another error following the success of importing the >>>>>> items: >>>>>> 11400 items imported so far... >>>>>> 11500 items imported so far... >>>>>> Total: 11505 items >>>>>> java.lang.NullPointerException >>>>>> at org.dspace.xoai.app.XOAI.willChangeStatus(XOAI.java:438) >>>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:368) >>>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:280) >>>>>> at org.dspace.xoai.app.XOAI.indexAll(XOAI.java:227) >>>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:134) >>>>>> at org.dspace.xoai.app.XOAI.main(XOAI.java:560) >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>> at >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>> at >>>>>> org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229) >>>>>> at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81) >>>>>> >>>>>> I did searches through the web and found that it will be resolved in >>>>>> 6.4... >>>>>> https://alanorth.github.io/cgspace-notes/2020-06/ >>>>>> >>>>>> Thanks a lot. >>>>>> Panyarak >>>>>> >>>>>> On Mon, Mar 22, 2021 at 11:16 PM FILIPPOS KOLOVOS <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Dear Sir, >>>>>>> >>>>>>> This error relates to the Garbage Collection (GC) mechanism of JVM, >>>>>>> which means that it does not have enough memory available to complete >>>>>>> the >>>>>>> task. More in particular, it means that the Garbage Collector is >>>>>>> spending >>>>>>> too much time clearing the heap space from unused objects, but at the >>>>>>> same >>>>>>> time it does not free more than 2% of the heap space, which is used for >>>>>>> the >>>>>>> instantiated objects. That's wh the GC reports that it does not have any >>>>>>> more space to do its job. >>>>>>> >>>>>>> One thing you can do, if you do not have enough RAM on your server, >>>>>>> is first shutdown Tomcat (this will lead to some downtime for your >>>>>>> server, >>>>>>> but it will free up some valuable RAM and after OAI completes you can >>>>>>> restart it) and then go to the */dspace/bin/dspace* file of your >>>>>>> running dspace instance and edit it with your text editor. Then, in the >>>>>>> file search and find the line: >>>>>>> >>>>>>> *JAVA_OPTS="-Xmx256m -Dfile.encoding=UTF-8"* >>>>>>> >>>>>>> and *change it *to: >>>>>>> >>>>>>> *JAVA_OPTS="-Xmx2048m -Xms1024m -Dfile.encoding=UTF-8"* >>>>>>> >>>>>>> *DO NOT* copy and paste the line, but edit it in place, because you >>>>>>> do not want to mess up this file. >>>>>>> >>>>>>> This will specify that the dspace *TOOL *(which is the file you are >>>>>>> running for the OAI) will use *from 1024MB up to 2048MB RAM* for >>>>>>> its task and not just *256MB*. However, you *must have more than >>>>>>> 2048MB RAM* available on the server (i.e. 3GB), or the script will >>>>>>> fail again and it might hang your server. If you have *LESS *than >>>>>>> 3GB allocated to the server, then adjust these values accordingly (i.e. >>>>>>> Xmx1024m Xms500m, etc). >>>>>>> >>>>>>> The bottomline is that you do not have enough memory to complete the >>>>>>> task and even if your server has 32GB RAM, the dspace tool will only >>>>>>> use up >>>>>>> to the RAM specified in this file. >>>>>>> >>>>>>> I hope that I have helped! >>>>>>> >>>>>>> Best Regards, >>>>>>> >>>>>>> -Fk >>>>>>> >>>>>>> On Mon, Mar 22, 2021 at 4:08 PM Panyarak Ngamsritragul < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am using DSpace 6.3+Apache Tomcat Version 8.0.37+javac 11.0.10 >>>>>>>> >>>>>>>> The instance I am managing has 11,428 records. (kb.psu.ac.th) >>>>>>>> >>>>>>>> I tried to create indexes for OAI using this command >>>>>>>> >>>>>>>> /dspace/bin/dspace oai import -c >>>>>>>> >>>>>>>> It worked until 8900 items and crashed with error messages: >>>>>>>> >>>>>>>> 8600 items imported so far... >>>>>>>> 8700 items imported so far... >>>>>>>> 8800 items imported so far... >>>>>>>> 8900 items imported so far... >>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>>>>> at java.util.Arrays.copyOfRange(Arrays.java:3664) >>>>>>>> at java.lang.String.<init>(String.java:207) >>>>>>>> at java.lang.StringBuilder.toString(StringBuilder.java:407) >>>>>>>> at >>>>>>>> org.hibernate.persister.entity.AbstractEntityPersister.selectFragment(AbstractEntityPersister.java:1422) >>>>>>>> at >>>>>>>> org.hibernate.persister.entity.AbstractEntityPersister.selectFragment(AbstractEntityPersister.java:4434) >>>>>>>> at >>>>>>>> org.hibernate.loader.JoinWalker.selectString(JoinWalker.java:1099) >>>>>>>> at >>>>>>>> org.hibernate.loader.AbstractEntityJoinWalker.initStatementString(AbstractEntityJoinWalker.java:123) >>>>>>>> at >>>>>>>> org.hibernate.loader.AbstractEntityJoinWalker.initStatementString(AbstractEntityJoinWalker.java:108) >>>>>>>> at >>>>>>>> org.hibernate.loader.AbstractEntityJoinWalker.initAll(AbstractEntityJoinWalker.java:90) >>>>>>>> at >>>>>>>> org.hibernate.loader.AbstractEntityJoinWalker.initAll(AbstractEntityJoinWalker.java:77) >>>>>>>> at >>>>>>>> org.hibernate.loader.criteria.CriteriaJoinWalker.<init>(CriteriaJoinWalker.java:123) >>>>>>>> at >>>>>>>> org.hibernate.loader.criteria.CriteriaJoinWalker.<init>(CriteriaJoinWalker.java:92) >>>>>>>> at >>>>>>>> org.hibernate.loader.criteria.CriteriaLoader.<init>(CriteriaLoader.java:95) >>>>>>>> at >>>>>>>> org.hibernate.internal.SessionImpl.list(SessionImpl.java:1604) >>>>>>>> at >>>>>>>> org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:374) >>>>>>>> at >>>>>>>> org.dspace.core.AbstractHibernateDAO.list(AbstractHibernateDAO.java:158) >>>>>>>> at >>>>>>>> org.dspace.authorize.dao.impl.ResourcePolicyDAOImpl.findByDSoAndAction(ResourcePolicyDAOImpl.java:74) >>>>>>>> at >>>>>>>> org.dspace.authorize.ResourcePolicyServiceImpl.find(ResourcePolicyServiceImpl.java:103) >>>>>>>> at >>>>>>>> org.dspace.authorize.AuthorizeServiceImpl.getPoliciesActionFilter(AuthorizeServiceImpl.java:575) >>>>>>>> at >>>>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorize(AuthorizeServiceImpl.java:301) >>>>>>>> at >>>>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeAction(AuthorizeServiceImpl.java:129) >>>>>>>> at >>>>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeAction(AuthorizeServiceImpl.java:95) >>>>>>>> at >>>>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeActionBoolean(AuthorizeServiceImpl.java:181) >>>>>>>> at >>>>>>>> org.dspace.authorize.AuthorizeServiceImpl.authorizeActionBoolean(AuthorizeServiceImpl.java:166) >>>>>>>> at org.dspace.xoai.app.XOAI.isPublic(XOAI.java:458) >>>>>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:343) >>>>>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:280) >>>>>>>> at org.dspace.xoai.app.XOAI.indexAll(XOAI.java:227) >>>>>>>> at org.dspace.xoai.app.XOAI.index(XOAI.java:134) >>>>>>>> at org.dspace.xoai.app.XOAI.main(XOAI.java:560) >>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>> at >>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>>> >>>>>>>> Could you please help to solve this problem? >>>>>>>> >>>>>>>> Thanks and best regards, >>>>>>>> Panyarak Ngamsritragul >>>>>>>> >>>>>>>> -- >>>>>>>> All messages to this mailing list should adhere to the Code of >>>>>>>> Conduct: https://duraspace.org/about/policies/code-of-conduct/ >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "DSpace Community" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/dspace-community/fcc5dfb5-0c01-4665-b490-e915d4662f96n%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/dspace-community/fcc5dfb5-0c01-4665-b490-e915d4662f96n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>> >>>>>>> -- All messages to this mailing list should adhere to the Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/ --- You received this message because you are subscribed to the Google Groups "DSpace Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-community/CAJc8HTea_S%2B8H18xeTy%3D4bPB_EqHfDM_J-Hkq6J6%3DQ1nrnT_-Q%40mail.gmail.com.
