Hmm, in our case of running gluster across Linode block storage (which itself runs inside Ceph, as I found out), the only thing that helped with the hangs so far was defragmenting xfs.
I tried changing many things, including the scheduler to "none" and this performance.write-behind-window-size setting, and nothing seemed to help or provide any meaningful difference. Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | @ArtemR <http://twitter.com/ArtemR> On Fri, Aug 7, 2020 at 11:28 AM Computerisms Corporation < b...@computerisms.ca> wrote: > Hi Artem and others, > > Happy to report the system has been relatively stable for the remainder > of the week. I have one wordpress site that seems to get hung processes > when someone logs in with an incorrect password. Since it is only one, > and reliably reproduceable, I am not sure if the issue is to do with > Gluster or Wordpress itself, but afaik it was not doing it some months > back before the system was using Gluster so I am guessing some combo of > both. > > Regardless, that is the one and only time apache processes stacked up to > over 150, and that still only brought the load average up to just under > 25; the system did go a bit sluggish, but remained fairly responsive > throughout until I restarted apache. Otherwise 15 minute load average > consistently runs between 8 and 11 during peak hours and between 4 and 7 > during off hours, and other than the one time I have not seen the > one-minute load average go over 15. all resources still spike to full > capacity from time to time, but it never remains that way for long like > it did before. > > For site responsiveness, first visit to any given site is quite slow, > like 3-5 seconds on straight html pages, 10-15 seconds for some of the > more bloated WP themes, but clicking links within the site after the > first page is loaded is relatively quick, like 1 second on straight html > pages, and ~5-6 seconds on the bloated themes. Again, not sure if that > is a Gluster related thing or something else. > > So, still holding my breath a bit, but seems this solution is working, > at least for me. I haven't played with any of the other settings yet to > see if I can improve it further, probably will next week. thinking to > increase the write behind window size further to see what happens, as > well as play with the settings suggested by Strahil. > > On 2020-08-05 5:28 p.m., Artem Russakovskii wrote: > > I'm very curious whether these improvements hold up over the next few > > days. Please report back. > > > > Sincerely, > > Artem > > > > -- > > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > > <http://www.apkmirror.com/>, Illogical Robot LLC > > beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR> > > > > > > On Wed, Aug 5, 2020 at 9:44 AM Computerisms Corporation > > <b...@computerisms.ca <mailto:b...@computerisms.ca>> wrote: > > > > Hi List, > > > > > So, we just moved into a quieter time of the day, but maybe I just > > > stumbled onto something. I was trying to figure out if/how I > could > > > throw more RAM at the problem. gluster docs says write behind is > > not a > > > cache unless flush-behind is on. So seems that is a way to throw > > ram to > > > it? I put performance.write-behind-window-size: 512MB and > > > performance.flush-behind: on and the whole system calmed down > pretty > > > much immediately. could be just timing, though, will have to see > > > tomorrow during business hours whether the system stays at a > > reasonable > > > load. > > > > so reporting back that this seems to have definitely had a > significant > > positive effect. > > > > So far today I have not seen the load average climb over 13 with the > > 15minute average hovering around 7. cpus are still spiking from > > time to > > time, but they are not staying maxed out all the time, and > frequently I > > am seeing brief periods of up to 80% idle. glusterfs process still > > spiking up to 180% or so, but consistently running around 70%, and > the > > brick processes still spiking up to 70-80%, but consistently running > > around 20%. Disk has only been above 50% in atop once so far today > > when > > it spiked up to 92%, and still lots of RAM left over. So far nload > > even > > seems indicates I could get away with a 100Mbit network connection. > > Websites are snappy relative to what they were, still a bit sluggish > on > > the first page of any given site, but tolerable or close to. Apache > > processes are opening and closing right away, instead of stacking up. > > > > Overall, system is performing pretty much like I would expect it to > > without gluster. I haven't played with any of the other settings > yet, > > just going to leave it like this for a day. > > > > I have to admit I am a little bit suspicious. I have been arguing > with > > Gluster for a very long time, and I have never known it to play this > > nice. kind feels like when your girl tells you she is "fine"; > > conversation has stopped, but you aren't really sure if it's done... > > > > > > > > I will still test the other options you suggested tonight, > > though, this > > > is probably too good to be true. > > > > > > Can't thank you enough for your input, Strahil, your help is truly > > > appreciated! > > > > > > > > > > > > > > > > > > > > >> > > >>>> > > >>>> > > >>>> Best Regards, > > >>>> Strahil Nikolov > > >>>> > > >>> ________ > > >>> > > >>> > > >>> > > >>> Community Meeting Calendar: > > >>> > > >>> Schedule - > > >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > > >>> Bridge: https://bluejeans.com/441850968 > > >>> > > >>> Gluster-users mailing list > > >>> Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> > > >>> https://lists.gluster.org/mailman/listinfo/gluster-users > > > ________ > > > > > > > > > > > > Community Meeting Calendar: > > > > > > Schedule - > > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > > > Bridge: https://bluejeans.com/441850968 > > > > > > Gluster-users mailing list > > > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > ________ > > > > > > > > Community Meeting Calendar: > > > > Schedule - > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > > Bridge: https://bluejeans.com/441850968 > > > > Gluster-users mailing list > > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> > > https://lists.gluster.org/mailman/listinfo/gluster-users > > >
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users