Re: [GTALUG] war story: gtalug.org's filled up
On Wed, Apr 05, 2023 at 06:24:22PM -0700, BCLUG via talk wrote: > > (Someone already replied to and addressed this, but since I'd already typed > it up, gonna send it anyway.) > > > o1bigtenor via talk wrote on 2023-04-05 12:34: > > > a great way > > to reduce the problem caused by a runaway var file was to use separate > > /var and /usr partitions (from / and /home). > > That *may* help with preserving space on /, but now you have 2 more > partitions to manage, and if /var runs out of space, you're still in > trouble. > > > > Plus, if /usr runs out of space, there'll be trouble installing anything. > > > Really, that just shifts the problem around without solving much. Also most linux distributions no longer support /usr being a different partition than / so those days are done. And it wasn't even a good idea when it was first introduced, but well they had to do what they could with the disk size they had available. -- Len Sorensen --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] war story: gtalug.org's filled up
(Someone already replied to and addressed this, but since I'd already typed it up, gonna send it anyway.) o1bigtenor via talk wrote on 2023-04-05 12:34: a great way to reduce the problem caused by a runaway var file was to use separate /var and /usr partitions (from / and /home). That *may* help with preserving space on /, but now you have 2 more partitions to manage, and if /var runs out of space, you're still in trouble. Plus, if /usr runs out of space, there'll be trouble installing anything. Really, that just shifts the problem around without solving much. My 2 --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] war story: gtalug.org's filled up
Antony mentioned missing reading mail with mutt. I’m still using mutt, and it works really well for me. It automatically formats HTML only messages (using the elinks command), can throw HTML at my browser when I ask, handles O365 multi-factor authentication, interprets calendar messages and so on. I can pipe messages into my folder filing command, filter and save bulk mail by rules, and various other things. I will admit it took a fair amount of setup over the years, and surely isn’t for everyone. But I find it makes my e-mail life easier in a number of ways. And I rarely have to click a mouse button. Cheers John --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] war story: gtalug.org's filled up
On 2023-04-05 15:34, o1bigtenor via talk wrote: On Wed, Apr 5, 2023 at 12:23 PM Kevin Cozens via talk wrote: ... To avoid the problem in the future I always have servers running logwatch. It gives me a daily report of system activity based on various log files. Of particular relevance to the topic of this message thread is the bit near the bottom. It reports on in use and available disk space. ... Daily reports getting mailed to you can have a problem that they're long and repetitive and get ignored after the novelty of reading them wears off. So long as they're concise and to the point and focus on anomalies and don't go listing all the normal things and you actually read them they can be ok. And some issues can go from zero to major problem in less than a 24-hour cycle so an alert sooner can help. Quite a number of years ago a now deceased mentor advised that a great way to reduce the problem caused by a runaway var file was to use separate /var and /usr partitions (from / and /home). (Not sure Thunderbird is getting the quotation levels right here, but the first bit was Kevin and this bit was o1bigtenor. Getting homesick for Mutt.) The filesystem root and /usr are static, so they only care at upgrade time that there's enough space then, and at other times will happily sail past filesystem-full events. Separation mostly makes sense if you want one thing to still work when another thing fills its partition, eg being able to write /var/log even though the dear users have filled /home or /tmp again. And nowadays /tmp is usually its own tmpfs already. Monitoring disk space and being proactive before usage hits the roof is a good thing for those of us who value tranquillity and hate having to deal with things like this at inconvenient times or after the users notice. But a long-ago coworker once said "We've got to be reactive, not proactive" because we were in an environment where you got brownie points for heroically responding to disk-full tickets, but if you kept things running smoothly you'd be looking at layoff papers. (In other news, I'm no longer there.) Anthony --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] war story: gtalug.org's filled up
Way back in October 2000, on the SAGE members mailing list, well-known expert Tom Limoncelli said: "It isn't a service if it isn't monitored. If there is no monitoring then you're just running software." --- Tom Limoncelli On Wed, 2023/04/05 02:34:15PM -0500, o1bigtenor via talk wrote: | Quite a number of years ago a now deceased mentor advised that a great way | to reduce the problem caused by a runaway var file was to use separate | /var and /usr partitions (from / and /home). I would argue that these days, in most cases, that is no longer a good configuration. Sure, /var filling up won't prevent you from manipulating files in /home. But if /var or / or /tmp is full, you've got a problem. And with multiple partitions, you need to have multiple chunks of free space. And you have to have a good idea up front of what your space requirements are. And yes, with LVM you can often extend a partition. But multiple partitions just mean more problems to worry about. Better you just alert once your disk gets 80% (or whatever) full. My two cents - cheers. John --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] war story: gtalug.org's filled up
On Wed, Apr 5, 2023 at 12:23 PM Kevin Cozens via talk wrote: > > On 2023-04-04 17:50, D. Hugh Redelmeier via talk wrote: > > GTALUG's server's filesystem filled up: little disk space left. > > I've had that happen to me in the past where the disk space on a server > suddenly runs out. One of the reasons it happened was due to the log file of > some program growing out of control. > > To avoid the problem in the future I always have servers running logwatch. > It gives me a daily report of system activity based on various log files. Of > particular relevance to the topic of this message thread is the bit near the > bottom. It reports on in use and available disk space. Assuming nothing > major happens in a day to fill up the space I can keep an eye on disk usage > and take action long before the amount of free space gets to a dangerously > low level. > > The other part of keeping an eye on things is running logrotate to keep the > size of log files under control. Both logwatch and logrotate run daily. > Quite a number of years ago a now deceased mentor advised that a great way to reduce the problem caused by a runaway var file was to use separate /var and /usr partitions (from / and /home). Together with the foregoing - - - - well you should just never have a problem. HTH --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] war story: gtalug.org's filled up
On 2023-04-04 17:50, D. Hugh Redelmeier via talk wrote: GTALUG's server's filesystem filled up: little disk space left. I've had that happen to me in the past where the disk space on a server suddenly runs out. One of the reasons it happened was due to the log file of some program growing out of control. To avoid the problem in the future I always have servers running logwatch. It gives me a daily report of system activity based on various log files. Of particular relevance to the topic of this message thread is the bit near the bottom. It reports on in use and available disk space. Assuming nothing major happens in a day to fill up the space I can keep an eye on disk usage and take action long before the amount of free space gets to a dangerously low level. The other part of keeping an eye on things is running logrotate to keep the size of log files under control. Both logwatch and logrotate run daily. -- Cheers! Kevin. https://www.patreon.com/KevinCozens | "Nerds make the shiny things that | distract the mouth-breathers, and Owner of Elecraft K2 #2172 | that's why we're powerful" #include | --Chris Hardwick --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
Re: [GTALUG] war story: gtalug.org's filled up
On Tue, 4 Apr 2023 at 17:50, D. Hugh Redelmeier via talk wrote: > > (This post is also a test of whether the mailing list is working.) > > GTALUG's server's filesystem filled up: little disk space left. > > I discovered this when I tried to do "apt update; apt full-upgrade". > The second step failed, saying that /var/cache/apt/archives had no > room. This means that / has no room because / contains that > directory. > > Tip: "df /some/path" will tell you how much space is used on the > filesystem containing /some/path and it will tell you the mount point > of that file system. > > I wandered around the filesystem, doing: > sudo df -s * | sort -n > > This command lists things in the current directory, and their sizes, > largest last. (It skips things with names starting with ".".) > > Pretty soon, I found that most of the space was taken up by > /var/cache/apt/archives after all. This kind of surprised me. > > Googling got me to others with this problem. The advice: > sudo apt autoclean > That gave back (only) 3% of the disk. > > That directory was still way too big. Most of the space was taken by > 35 versions of gitlab-runner. I have no idea why we need multiple > versions. > > Violence is sometimes the answer. > sudo apt clean > > That left only 32% of / used. > > Note /var/cache/apt/archives is only a cache. If the system wants any > of these, it should be able to find them in a repo. I used to use a method very similar to yours - I presume you meant 'du' not 'df'. I used 'du -sh * | sort -h' which I found easier to read and understand. But then - "Hallelujah" (let's make this a religious discovery) I found 'ncdu', which is a TUI that does exactly the same thing, but makes it explorable by selecting and entering any directory to see the same listing inside that new dir. Give it a try: it makes the whole process of finding space-hogging directories immensely faster. And because it's a TUI, not a GUI, it works on remote hosts over SSH even if they don't have X/Wayland. I don't know how long it's been in the Debian repos: it's in the current Debian, it may not be in older ones. -- Giles https://www.gilesorr.com/ giles...@gmail.com --- Post to this mailing list talk@gtalug.org Unsubscribe from this mailing list https://gtalug.org/mailman/listinfo/talk
