[dspace-tech] Re: DSpace 7 on production - poor performance
There is an architectural issue in the angular -> API design which means that a very large number of calls are made to the API for each page load. This also makes the logs very noisy. I have found 16GB lean for DSpace 7.5 where the database is on the same server. Tomcat needs about 3G, Each pm2 takes about 1G and SOLR and postgres chew up a lot. ClamAV daemon can chew up another 2G. You might want to *reduce* the number of pm2 instances as you may be running low on memory. If your system is starting to swap this can slow things down terribly. Adjust the robots to block entity paths and browse paths as robots can get lost in the DSpace search (a historical problem). The following can help show where traffic is coming from cat access.log | grep -v " 403 " | grep -v " 301 " grep -v " 408 " | cut -d " " -f 1 | sort | uniq -c | sort -n Unfortunately you will find that a lot of the traffic is to the API server but you can identify bots this way. fail2ban is a useful tool to block sim-behaving bots. IMHO DSpace7 needs a bit more work architecturally to improve performance.That is understandable, it is huge (and impressive) migration that has been completed from DSpace 6. The new version is a very fresh and nice design, and the new API is nice. Edmund Balnaves Prosentient Systems On Tuesday, June 27, 2023 at 5:57:38 AM UTC+10 Karol wrote: > Hi, > > I have implemented dspace7 in production, I have 4vCPU 16 Gb ram . I start > angular using all cpu pm2 -i max, but the performance of the whole site is > very bad. Now I can see my apache logs are growing fast: access.log and > dspace.log - probably bots are indexing new content and this is killing my > site, and real users can't submit work and use the repository. > Unfortunately, I can't tell 100% what or who is overloading the system, > because the apache logs show my server address (probably by using a proxy > for angular) > > * "top" shows 130% CPU 20% ram node > /dspace-angular-7.5/dist/server/main.js - this is where I'm looking for a > performance problem. > > * Apache access.logs per day take up 400 MB - I see continuous logging, > but I can't tell from which IP addresses. The dspace log: dspace.log 300MB > per day. > > 1) How can I increase the performance of angular: node > /dspace-angular-7.5/dist/server/main.js ? (I already use pm2 -i max) > > 2) How can I check from which addresses are so many requests flying to > dspace7 ? > > > Thanks and best regards, > > Karol > -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/e86e92be-0577-4e9c-a851-fbeff5feb4bfn%40googlegroups.com.
[dspace-tech] DSpace 7 on production - poor performance
Hi, I have implemented dspace7 in production, I have 4vCPU 16 Gb ram . I start angular using all cpu pm2 -i max, but the performance of the whole site is very bad. Now I can see my apache logs are growing fast: access.log and dspace.log - probably bots are indexing new content and this is killing my site, and real users can't submit work and use the repository. Unfortunately, I can't tell 100% what or who is overloading the system, because the apache logs show my server address (probably by using a proxy for angular) * "top" shows 130% CPU 20% ram node /dspace-angular-7.5/dist/server/main.js - this is where I'm looking for a performance problem. * Apache access.logs per day take up 400 MB - I see continuous logging, but I can't tell from which IP addresses. The dspace log: dspace.log 300MB per day. 1) How can I increase the performance of angular: node /dspace-angular-7.5/dist/server/main.js ? (I already use pm2 -i max) 2) How can I check from which addresses are so many requests flying to dspace7 ? Thanks and best regards, Karol -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/f1440bbc-e74f-4cb2-9d6d-5b2bbeb54059n%40googlegroups.com.
Re: [Extern] [dspace-tech] Display streaming video on DSpace 7
Thank you Michael. On Monday, June 26, 2023 at 9:55:04 AM UTC-4 Michael Plate wrote: > Hi Joshua, > > Am 26.06.23 um 15:45 schrieb Joshua Kim: > > Hi, > > > > With DSpace 7.3, I can see mp4. file for download on the web page. But > > how can I make it display on the site instead of being downloaded? I > > have searched the documentation but I can't find it. > […] > > look into the config.yaml of dspace-angular…at the end you will find > settings "mediaViewer" for images and videos. > > CU > > Michael > -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/a0e0b9a9-3605-4197-a49e-e1b91a4eb578n%40googlegroups.com.
Re: [dspace-tech] If I can use Dspace to run a senarionof sharing data sets.
On Sun, Jun 25, 2023 at 04:50:21AM -0700, Alireza Mosavi Jarrahi wrote:
> I am wondering if Dspace might be the right tool for my purposes. Can
> someone tell me if the following scenario is supported by Dsapce:
>
> Person A has authored a data set, which he/she cannot easily disclose.
> Person A uploads the data set to Dspace, but as a private data set that
> others can only see the meta-data about.
I think this can be done by setting public access on the "Item"
(metadata and a group of files) and no access on the attached files
("Bitstream"s).
> Now Person B is interested in working with the data. He/she contacts Person
> A via a web-form in Dspace, requesting data access. Person A can now decide
> whether or not Person B should get access to the data set. If approved,
> Person B can access the full data on Dspace for a defined period. Dspace
> logs the entire interaction between the two Persons.
https://wiki.lyrasis.org/display/DSDOC7x/Request+a+Copy
This works by emailing a copy to the requester (if approved).
I do not know what is logged by the copy request code.
--
Mark H. Wood
Lead Technology Analyst
University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu
--
All messages to this mailing list should adhere to the Code of Conduct:
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dspace-tech/ZJmZLyJTFUQBjFCE%40IUPUI.Edu.
signature.asc
Description: PGP signature
Re: [Extern] [dspace-tech] Display streaming video on DSpace 7
Hi Joshua, Am 26.06.23 um 15:45 schrieb Joshua Kim: Hi, With DSpace 7.3, I can see mp4. file for download on the web page. But how can I make it display on the site instead of being downloaded? I have searched the documentation but I can't find it. […] look into the config.yaml of dspace-angular…at the end you will find settings "mediaViewer" for images and videos. CU Michael -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/eb04991f-354b-3ce9-a565-dca1ce3d4cec%40bibliothek.uni-kassel.de. smime.p7s Description: S/MIME Cryptographic Signature
[dspace-tech] Display streaming video on DSpace 7
Hi, With DSpace 7.3, I can see mp4. file for download on the web page. But how can I make it display on the site instead of being downloaded? I have searched the documentation but I can't find it. Joshua Kim Web Developer, Library IT Library, Museums and Press University of Delaware -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/CAOE6Kn5439gsTRvDS_NMwyiLJpNbJDy7V5vopLxw%2BbvjJD7c3Q%40mail.gmail.com.
[dspace-tech] Index fail after upgrading from Dspace 5.10 to 7.5
Dear DSpace-Tech Team, We upgraded our DSpace installation from 5.10 to 7.5 and everything went well, but while indexing we are getting the following error message. Kindly help. The script has started Updating Index java.lang.NullPointerException at org.dspace.discovery.indexobject.InprogressSubmissionIndexFactoryImpl.storeInprogressItemFields(InprogressSubmissionIndexFactoryImpl.java:61) at org.dspace.discovery.indexobject.InprogressSubmissionIndexFactoryImpl.buildDocument(InprogressSubmissionIndexFactoryImpl.java:53) at org.dspace.discovery.indexobject.WorkflowItemIndexFactoryImpl.buildDocument(WorkflowItemIndexFactoryImpl.java:73) at org.dspace.discovery.indexobject.WorkflowItemIndexFactoryImpl.buildDocument(WorkflowItemIndexFactoryImpl.java:36) at org.dspace.discovery.SolrServiceImpl.update(SolrServiceImpl.java:169) at org.dspace.discovery.SolrServiceImpl.indexContent(SolrServiceImpl.java:159) at org.dspace.discovery.SolrServiceImpl.updateIndex(SolrServiceImpl.java:344) at org.dspace.discovery.SolrServiceImpl.updateIndex(SolrServiceImpl.java:331) at org.dspace.discovery.IndexClient.internalRun(IndexClient.java:137) at org.dspace.scripts.DSpaceRunnable.run(DSpaceRunnable.java:104) at org.dspace.app.launcher.ScriptLauncher.executeScript(ScriptLauncher.java:149) at org.dspace.app.launcher.ScriptLauncher.handleScript(ScriptLauncher.java:131) at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:98) Regards Maruthu Pandiyan The Institute of Mathematical Sciences, Chennai -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/7af7a234-01fa-47bf-967a-4778eac3a93bn%40googlegroups.com.
[dspace-tech] Forgot password errors
Dear All, When resetting pwd on DSpace 7.4 I am getting error, see attached postgres log and frontend screenshot error We recently migrated our DSpace server to Oracle cloud and this errors started to show, when users tries to reset their passwd they receive attached error details. Regards, Lewatle -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/1a8bff52--42cd-ae74-0e94e0f4fcd6n%40googlegroups.com. $ vi postgresql-11-main.log 2023-06-20 17:27:51.429 UTC [378729] wiredmin@wiredspace DETAIL: Key (email)=(0...@students.wits.ac.za) already exists. 2023-06-20 17:27:51.429 UTC [378729] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where registrationdata_id=$4 2023-06-20 17:27:55.528 UTC [378727] wiredmin@wiredspace ERROR: duplicate key value violates unique constraint "registrationdata_email_key" 2023-06-20 17:27:55.528 UTC [378727] wiredmin@wiredspace DETAIL: Key (email)=(0...@students.wits.ac.za) already exists. 2023-06-20 17:27:55.528 UTC [378727] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where registrationdata_id=$4 2023-06-22 23:25:48.812 UTC [404261] wiredmin@wiredspace ERROR: duplicate key value violates unique constraint "registrationdata_email_key" 2023-06-22 23:25:48.812 UTC [404261] wiredmin@wiredspace DETAIL: Key (email)=(9...@students.wits.ac.za) already exists. 2023-06-22 23:25:48.812 UTC [404261] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where registrationdata_id=$4 2023-06-22 23:25:52.905 UTC [404216] wiredmin@wiredspace ERROR: duplicate key value violates unique constraint "registrationdata_email_key" 2023-06-22 23:25:52.905 UTC [404216] wiredmin@wiredspace DETAIL: Key (email)=(9...@students.wits.ac.za) already exists. 2023-06-22 23:25:52.905 UTC [404216] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where registrationdata_id=$4 2023-06-22 23:26:09.322 UTC [404174] wiredmin@wiredspace ERROR: duplicate key value violates unique constraint "registrationdata_email_key" 2023-06-22 23:26:09.322 UTC [404174] wiredmin@wiredspace DETAIL: Key (email)=(9...@students.wits.ac.za) already exists. 2023-06-22 23:26:09.322 UTC [404174] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where registrationdata_id=$4 2023-06-22 23:26:09.324 UTC [404269] wiredmin@wiredspace ERROR: duplicate key value violates unique constraint "registrationdata_email_key" 2023-06-22 23:26:09.324 UTC [404269] wiredmin@wiredspace DETAIL: Key (email)=(9...@students.wits.ac.za) already exists. 2023-06-22 23:26:09.324 UTC [404269] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where registrationdata_id=$4 2023-06-22 23:26:11.333 UTC [404199] wiredmin@wiredspace ERROR: duplicate key value violates unique constraint "registrationdata_email_key" 2023-06-22 23:26:11.333 UTC [404199] wiredmin@wiredspace DETAIL: Key (email)=(9...@students.wits.ac.za) already exists. 2023-06-22 23:26:11.333 UTC [404199] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where registrationdata_id=$4 2023-06-22 23:26:40.004 UTC [404273] wiredmin@wiredspace ERROR: duplicate key value violates unique constraint "registrationdata_email_key" 2023-06-22 23:26:40.004 UTC [404273] wiredmin@wiredspace DETAIL: Key (email)=(9...@students.wits.ac.za) already exists. 2023-06-22 23:26:40.004 UTC [404273] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where registrationdata_id=$4 2023-06-22 23:26:50.247 UTC [404277] wiredmin@wiredspace ERROR: duplicate key value violates unique constraint "registrationdata_email_key" 2023-06-22 23:26:50.247 UTC [404277] wiredmin@wiredspace DETAIL: Key (email)=(9...@students.wits.ac.za) already exists. 2023-06-22 23:26:50.247 UTC [404277] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where registrationdata_id=$4 2023-06-22 23:26:52.292 UTC [404268] wiredmin@wiredspace ERROR: duplicate key value violates unique constraint "registrationdata_email_key" 2023-06-22 23:26:52.292 UTC [404268] wiredmin@wiredspace DETAIL: Key (email)=(9...@students.wits.ac.za) already exists. 2023-06-22 23:26:52.292 UTC [404268] wiredmin@wiredspace STATEMENT: update public.registrationdata set email=$1, expires=$2, token=$3 where
