And anyways, whole bitstreams shouldn't be in DSpace apps memory, no? - Anis
On Friday, November 4, 2016 at 4:23:05 PM UTC+2, Anis wrote: > > Yes I understand that, but the image above is from a live process, and > there was no attack happening (this is confirmed), so for some reason there > is 5 gigs of byte arrays in the memory, tomcats uptime is about 16 hours. > > - Anis > > On Friday, November 4, 2016 at 3:48:13 PM UTC+2, Luiz dos Santos wrote: >> >> Hi Anis, >> >> This seems like you have a lot of bytes "[B" in memory, bitstream I >> guess, maybe this guys stay in the memory all the time because the bot is >> calling them over and over again. I already got have some bots that make 3 >> or 5 request per second in the same item, you should try to figure out in >> DSpace log the most frequently ip's. You should see in the DSpace log some >> exception saying that the database is running out of connection as well. >> If you got the DSpace UI frozen, the restart of tomcat can help. >> >> Best regards >> Luiz >> >> Luiz Claudio Santos >> http://luizclaudiosantos.me/ >> >> On Fri, Nov 4, 2016 at 8:16 AM, Anis <[email protected]> wrote: >> >>> We thought so too and we started blocking known violators. The >>> attachment shows the live heap map, and the byte arrays are still >>> dominating, taking 5 out of the 8Gb heap space, these are probably >>> responses to requests, but why are not they GC'd and why are they staying >>> in the memory? Maybe this is normal behaviour, so some clarification would >>> be nice. >>> >>> - Anis >>> >>> >>> <https://lh3.googleusercontent.com/-SpHRfW0dOmc/WBx6-jx2U4I/AAAAAAAABcc/pKF_EwUFsRkyvfODV5xVdi5Gok7Maxe6ACLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B14.05.58.png> >>> >>> >>> On Friday, November 4, 2016 at 2:08:34 PM UTC+2, Luiz dos Santos wrote: >>>> >>>> Hi Anis, >>>> >>>> To me it seems more like a bot attack then a leak of memory, do you >>>> have a strategy to deal with malicious bots? >>>> >>>> Best >>>> Luiz >>>> >>>> On Friday, November 4, 2016, Anis <[email protected]> wrote: >>>> >>>>> Hi! >>>>> >>>>> >>>>> Our busiest repository was crashing between every 15-30 minutes and it >>>>> seems it was related to out of memory problems and the amount of requests >>>>> DSpace is getting. We captured the heap dump from one of our crashes, >>>>> and as you can see from the image, 78% (3.12GB) of the heap space is >>>>> populated by byte arrays. A lot of these byte arrays are buffered 1MB >>>>> arrays that are part of an outputstream from >>>>> HttpServletResponseBufferingWrapper, and to my understanding this class >>>>> should send the 404 response and if you actually check the contents of >>>>> the >>>>> arrays, they have the 404 HTML code. It also seems that the >>>>> HttpServletResponseBufferingWrappers, that are in the memory are never >>>>> sent, as their response bytesWritten are all 0. >>>>> >>>>> We resolved the problem by letting Apache handle most of the 404, >>>>> cleaning our theme files from references, that are not found anymore, >>>>> capping out maxdb connections to 50 and maxidle to 30. It is possible, >>>>> that >>>>> we just had way too many 404 responses to serve, but it would be nice to >>>>> hear if someone else has had or is having this problem and if it happens >>>>> because of the amount of requests, it would be nice to know why. >>>>> >>>>> >>>>> - Anis >>>>> >>>>> >>>>> >>>>> >>>>> <https://lh3.googleusercontent.com/-6wbDPcg8vmA/WBxkRWhH0vI/AAAAAAAABb0/MlSZNTtzKO4YvgUI97FYfGmFsOnu1pY0gCLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B12.00.52.png> >>>>> >>>>> >>>>> <https://lh3.googleusercontent.com/-gXnStN9z8EE/WBxkWKq94EI/AAAAAAAABb4/WsBFZI-JxBQpHNMn21IBLY49yVshIxaNgCLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B12.01.12.png> >>>>> >>>>> >>>>> >>>>> <https://lh3.googleusercontent.com/-Ah_IQ4sWPxg/WBxkpB0ModI/AAAAAAAABb8/LA5PpCDhebIINgBxSY6Aj5AjGEpFV6IyQCLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B12.12.05.png> >>>>> >>>>> >>>>> <https://lh3.googleusercontent.com/-kcffJx_9HQY/WBxo7ypMrgI/AAAAAAAABcM/w8DSsqhop9kZgk9p31mxQHHgnDLTXKPsQCLcB/s1600/Screen%2BShot%2B2016-11-04%2Bat%2B12.53.12.png> >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "DSpace Technical Support" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/dspace-tech. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
