Hi, there are a number of issues for you to look into: https://jira.duraspace.org/browse/DS-2832 https://jira.duraspace.org/browse/DS-2869
(both fixed in DSpace 5.4) https://jira.duraspace.org/browse/DS-2788 (not yet fixed, but kind of ephemeral) rambling thoughts on tuning Solr: http://hardyoyo.thebignow.com/article/49/data-analysis-and-server-optimization-yesterday-was-a-good-day and a way to tame that aggressive web crawler: http://hardyoyo.thebignow.com/article/50/manage-all-web-crawler-sessions-like-user-sessions-yes-please --Hardy On Fri, Nov 4, 2016 at 10:30 AM, Luiz dos Santos <[email protected]> wrote: > Hi Anis, > > Sorry, I can't help you, but I am curious about what Hardy have to > say. > > Best regards > Luiz Claudio Santos > http://luizclaudiosantos.me/ > > On Fri, Nov 4, 2016 at 11:17 AM, Hardy Pottinger < > [email protected]> wrote: > >> This is starting to sound familiar. What version of DSpace are you >> running? >> >> --Hardy >> >> On Fri, Nov 4, 2016 at 9:29 AM, Anis <[email protected]> wrote: >> >>> And anyways, whole bitstreams shouldn't be in DSpace apps memory, no? >>> >>> - Anis >>> >>> >>> On Friday, November 4, 2016 at 4:23:05 PM UTC+2, Anis wrote: >>>> >>>> Yes I understand that, but the image above is from a live process, and >>>> there was no attack happening (this is confirmed), so for some reason there >>>> is 5 gigs of byte arrays in the memory, tomcats uptime is about 16 hours. >>>> >>>> - Anis >>>> >>>> On Friday, November 4, 2016 at 3:48:13 PM UTC+2, Luiz dos Santos wrote: >>>>> >>>>> Hi Anis, >>>>> >>>>> This seems like you have a lot of bytes "[B" in memory, bitstream >>>>> I guess, maybe this guys stay in the memory all the time because the bot >>>>> is >>>>> calling them over and over again. I already got have some bots that make 3 >>>>> or 5 request per second in the same item, you should try to figure out in >>>>> DSpace log the most frequently ip's. You should see in the DSpace log some >>>>> exception saying that the database is running out of connection as well. >>>>> If you got the DSpace UI frozen, the restart of tomcat can help. >>>>> >>>>> Best regards >>>>> Luiz >>>>> >>>>> Luiz Claudio Santos >>>>> http://luizclaudiosantos.me/ >>>>> >>>>> On Fri, Nov 4, 2016 at 8:16 AM, Anis <[email protected]> wrote: >>>>> >>>>>> We thought so too and we started blocking known violators. The >>>>>> attachment shows the live heap map, and the byte arrays are still >>>>>> dominating, taking 5 out of the 8Gb heap space, these are probably >>>>>> responses to requests, but why are not they GC'd and why are they staying >>>>>> in the memory? Maybe this is normal behaviour, so some clarification >>>>>> would >>>>>> be nice. >>>>>> >>>>>> - Anis >>>>>> >>>>>> >>>>>> <https://lh3.googleusercontent.com/-SpHRfW0dOmc/WBx6-jx2U4I/AAAAAAAABcc/pKF_EwUFsRkyvfODV5xVdi5Gok7Maxe6ACLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B14.05.58.png> >>>>>> >>>>>> >>>>>> On Friday, November 4, 2016 at 2:08:34 PM UTC+2, Luiz dos Santos >>>>>> wrote: >>>>>>> >>>>>>> Hi Anis, >>>>>>> >>>>>>> To me it seems more like a bot attack then a leak of memory, do >>>>>>> you have a strategy to deal with malicious bots? >>>>>>> >>>>>>> Best >>>>>>> Luiz >>>>>>> >>>>>>> On Friday, November 4, 2016, Anis <[email protected]> wrote: >>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> >>>>>>>> Our busiest repository was crashing between every 15-30 minutes and >>>>>>>> it seems it was related to out of memory problems and the amount of >>>>>>>> requests DSpace is getting. We captured the heap dump from one of >>>>>>>> our crashes, and as you can see from the image, 78% (3.12GB) of the >>>>>>>> heap >>>>>>>> space is populated by byte arrays. A lot of these byte arrays are >>>>>>>> buffered >>>>>>>> 1MB arrays that are part of an outputstream from >>>>>>>> HttpServletResponseBufferingWrapper, and to my understanding this >>>>>>>> class should send the 404 response and if you actually check the >>>>>>>> contents >>>>>>>> of the arrays, they have the 404 HTML code. It also seems that the >>>>>>>> HttpServletResponseBufferingWrappers, that are in the memory are >>>>>>>> never sent, as their response bytesWritten are all 0. >>>>>>>> >>>>>>>> We resolved the problem by letting Apache handle most of the 404, >>>>>>>> cleaning our theme files from references, that are not found anymore, >>>>>>>> capping out maxdb connections to 50 and maxidle to 30. It is possible, >>>>>>>> that >>>>>>>> we just had way too many 404 responses to serve, but it would be nice >>>>>>>> to >>>>>>>> hear if someone else has had or is having this problem and if it >>>>>>>> happens >>>>>>>> because of the amount of requests, it would be nice to know why. >>>>>>>> >>>>>>>> >>>>>>>> - Anis >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> <https://lh3.googleusercontent.com/-6wbDPcg8vmA/WBxkRWhH0vI/AAAAAAAABb0/MlSZNTtzKO4YvgUI97FYfGmFsOnu1pY0gCLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B12.00.52.png> >>>>>>>> >>>>>>>> >>>>>>>> <https://lh3.googleusercontent.com/-gXnStN9z8EE/WBxkWKq94EI/AAAAAAAABb4/WsBFZI-JxBQpHNMn21IBLY49yVshIxaNgCLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B12.01.12.png> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> <https://lh3.googleusercontent.com/-Ah_IQ4sWPxg/WBxkpB0ModI/AAAAAAAABb8/LA5PpCDhebIINgBxSY6Aj5AjGEpFV6IyQCLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B12.12.05.png> >>>>>>>> >>>>>>>> >>>>>>>> <https://lh3.googleusercontent.com/-kcffJx_9HQY/WBxo7ypMrgI/AAAAAAAABcM/w8DSsqhop9kZgk9p31mxQHHgnDLTXKPsQCLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B12.53.12.png> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "DSpace Technical Support" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/dspace-tech. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "DSpace Technical Support" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/dspace-tech. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "DSpace Technical Support" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/dspace-tech. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
