Re: swap_pager complaints but not using swap
In message 200901252247.29775.fbsd.questi...@rachie.is-a-geek.net, Mel writes: On Sunday 25 January 2009 04:28:47 Dieter wrote: AMD64 FreeBSD 7.0 2 GiB main memory So the machine doesn't normally use swap much at all, but messing with the large ISO apparently kicked something out of memory, and the disk with the swap partition was already busy writing at the other end of the disk. Do you consider writing a large file to disk a truly unspeakable load? Just curious, since you're running 7.0: SCHED_4BSD or SCHED_ULE? And if you have a chance to change it, does the scenario persist? kern.sched.name: 4BSD Right now I'm trying to collect enough info to fix the Seagate 2700.11 firmware fubar. Once that is resolved, I plan to update to 7.1. IIRC 7.1 defaults to ULE. If not I can try switching. But my understanding is that the scheduler pays little if any attention to I/O, a holdover from the days when CPU was the main thing that needed rationing. These days it is just the opposite. CPUs have gotten much much faster, but I/O is only slightly faster. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: swap_pager complaints but not using swap
Dieter wrote: AMD64 FreeBSD 7.0 2 GiB main memory My console says: login: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 pstat -sk Device 1K-blocks UsedAvail Capacity /dev/ad6s10 4590208 96 4590112 0% Wow, using a whole 96K of swap. I don't see any disk related complaints in dmesg. Is this something to worry about? Yes, the system was *trying* to do swap I/O and timing out while doing so. Kris Whoops, I forgot to change the subject line after adding the k option to pstat. Without the k it said 0 used. And this morning it occurs to me that even if swap used was zero, it could have been trying to *start* using swap. Anyway... given this timeout explaination, I'm guessing that page/swap has to compete with user processes for disk i/o, and thus probably suffers from the same lack of fair i/o scheduling that user processes suffer from. E.g. one process doing disk i/o can lock out another process for at least several minutes, probably indefinitely. :-( There is a timeout of (from memory) 60 seconds. I've not seen this timeout exceeded on properly functioning disk hardware (even heavily loaded), only on broken hardware/controllers, or on I/O devices that are intrinsically slow for some reason (USB stick, or swapping to a file). Unless you're doing something truly unspeakable to that disk's load, I'd look at the hardware. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: swap_pager complaints but not using swap
On Sat, 24 Jan 2009 14:48:39 + Dieter free...@sopwith.solgatos.com wrote: AMD64 FreeBSD 7.0 2 GiB main memory My console says: login: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 pstat -sk Device 1K-blocks UsedAvail Capacity /dev/ad6s10 4590208 96 4590112 0% Wow, using a whole 96K of swap. I don't see any disk related complaints in dmesg. Is this something to worry about? Yes, the system was *trying* to do swap I/O and timing out while doing so. isn't swapspace supposed to be on a 'b' partition? Are you using swap on a slice 10? how is that possible when the i386/amd64 BIOS can't see more than 4 primary partitions? Kris, would you mind giving input to this? How can there be a s10, and how can you add swapspace to a device that isn't a partition 'b' nor a file backed swapspace? Those were the only two ways I thought was supported for swap. Dieter, does my questions above sound to be a correct interpretation of your disk setup? Traditionally swap used the b partition. But then traditionally, there Even then, however, the swapdev line in the kernel config file made it possible to change it and to add additional partitions of equal size on other drives. weren't MBR style partitions, called slices in FreeBSD-land. I suspect that the computers Unix grew up on (PDP-7, PDP-11, VAX) had to boot from the beginning of the disk, so the a partition went there. The Alpha continues in this DEC tradition. I was about to say that swap went next for speed, since the machines back then never had enough main memory, but those old disks didn't have variable number of sectors on inner vs outer tracks, so the speed would have been the same across the platter. So I'm not sure why swap was next. Speed was indeed the reason. Swapping in UNIX dates further back than virtual memory support. Swapping made it possible to have more process memory allocated than was available in real memory, but it was necessary to swap out an entire process to swap in a process whose memory was located at the same addresses as the one forced out. That didn't necessarily mean writing all of the first process's memory out, just enough to make room for the one coming in. The downside, of course, was that anything *not* written to the swap area was vulnerable to damage by other processes unless some sort of storage protection mechanism was available in the hardware a la System/360 storage protection. Of course, the same risk applied to any other processes in memory (i.e., not swapped out already) at the time, too. Swap in 3BSD (the first VM UNIX system) and 4BSD was active for another purpose, too. Large memory moves/copies on those old machines could actually be *slower* than writing to a disk and then reading the data back in at the target memory location. (Yes, speechless horror is the correct reaction here.:-) /usr was typically placed right after the swap area (i.e., partition d). The idea was that / and /usr would be the most heavily accessed file systems, so it made sense to have them surrounding the swap area to minimize head movement distances and delays. On systems with more than one disk drive that did a lot of compiling, sorting, editing, or anything else with much activity in /tmp, it was especially helpful to move /tmp to a separate drive from / and /usr. The manual used to give suggested partition configurations for two- and three-drive systems for optimum speed. This machine has 2 GiB of main memory and almost never uses the swap partition, so I put swap at the slow end of the drive. Yes I have swap on slice 10. I use NetBSD's fdisk, as it handles more than 4 slices nicely, unlike FreeBSD's fdisk. As far as I know, the BIOS So NetBSD's fdisk understands logical partitions in an extended partition? Cool. I wish we had it in FreeBSD. It's really a pain to have to shut FreeBSD down and boot a standalone program to change the layout of a disk that has an EP. :-( At least the FreeBSD kernel has no problem understanding a disk like that. firmware doesn't need to know about swap. I think the BIOS firmware just loads and runs the MBR, which in turn loads and runs the bootstrap in the selected slice (or loads and runs the MBR in a different disk if you want). I suppose I could put a BSD disklabel on slice 10 and set it up with the whole slice as the b partition. But as far as I can tell FreeBSD is happy with /dev/ad6s10. As I wrote in my previous It should be. message I suspect that the pager/swaper is competing for disk i/o. I forgot to ask if there is some sysctl or other knob to lengthen the timeout. The real fix is to improve the i/o fairness,
Re: swap_pager complaints but not using swap
AMD64 FreeBSD 7.0 2 GiB main memory My console says: login: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 pstat -sk Device 1K-blocks UsedAvail Capacity /dev/ad6s10 4590208 96 4590112 0% Wow, using a whole 96K of swap. I don't see any disk related complaints in dmesg. Is this something to worry about? Yes, the system was *trying* to do swap I/O and timing out while doing so. Kris Whoops, I forgot to change the subject line after adding the k option to pstat. Without the k it said 0 used. And this morning it occurs to me that even if swap used was zero, it could have been trying to *start* using swap. Anyway... given this timeout explaination, I'm guessing that page/swap has to compete with user processes for disk i/o, and thus probably suffers from the same lack of fair i/o scheduling that user processes suffer from. E.g. one process doing disk i/o can lock out another process for at least several minutes, probably indefinitely. :-( There is a timeout of (from memory) 60 seconds. I've not seen this timeout exceeded on properly functioning disk hardware (even heavily loaded), only on broken hardware/controllers, or on I/O devices that are intrinsically slow for some reason (USB stick, or swapping to a file). Unless you're doing something truly unspeakable to that disk's load, I'd look at the hardware. zcat /ad8/7.1-RELEASE-amd64-dvd1.iso.gz /ad6/7.1-RELEASE-amd64-dvd1.iso I'll spare you the real paths. :-) The target was to slice 2, which is very near the beginning of the disk, while swap is in slice 10 at the very end of the same disk. These disks are both Seagate 7200 SATA connected to nforce4-ultra. I just ran the same command again, and the CPU is 89-96% idle. So it is I/O bound writing to the disk, as expected. The machine was rebooted Tuesday afternoon (I had been testing a firewire patch for Sean). Friday morning I copied the 7.1 ISO to the machine and was verifying checksums. After I noticed the swap_pager complaints on the console I checked and it was only using 96 KiB of swap. Two days later (Sunday morning) swap usage has grown to 500 KiB: pstat -sk Device 1K-blocks UsedAvail Capacity /dev/ad6s10 4590208 500 4589708 0% So the machine doesn't normally use swap much at all, but messing with the large ISO apparently kicked something out of memory, and the disk with the swap partition was already busy writing at the other end of the disk. Do you consider writing a large file to disk a truly unspeakable load? Scott Bennett writes: This machine has 2 GiB of main memory and almost never uses the swap partition, so I put swap at the slow end of the drive. Yes I have swap on slice 10. I use NetBSD's fdisk, as it handles more than 4 slices nicely, unlike FreeBSD's fdisk. As far as I know, the BIOS So NetBSD's fdisk understands logical partitions in an extended partition? Cool. I wish we had it in FreeBSD. It's really a pain to have to shut FreeBSD down and boot a standalone program to change the layout of a disk that has an EP. :-( At least the FreeBSD kernel has no problem understanding a disk like that. I haven't ported NetBSD's fdisk to FreeBSD, I just boot NetBSD, fdisk the new disk, and boot back to FreeBSD. I also use NetBSD's MBR, which has a nice boot menu. (well, as nice as it can be with only 512 bytes to work with) It would be nice to have NetBSD's fdisk ported, as FreeBSD's fdisk can't even read the logical/extended partitions. message I suspect that the pager/swaper is competing for disk i/o. I forgot to ask if there is some sysctl or other knob to lengthen the timeout. The real fix is to improve the i/o fairness, but I've been asking about this for 2-3 years and not getting anywhere. BSD UNIX introduced the disksort() routine into its kernel ages ago. I know it was in 4.2BSD, but it may well have been there long before then. disksort() was added to satisfy a maximum number of disk I/O requests with a minimum of head movement and delay. Basically, it sorts new requests into queues for each drive such that the arm moves from request to request in one direction through the disk, and then the next queue started is sorted into the opposite sequence for the arm to move in the opposite direction. The result is that the arm moves back and forth from the start to the end of the disk and then back again, reading and writing as it goes, thus minimizing the distance traveled for each request handled. In FreeBSD, I think there is also some sort of change to the algorithm that tends to subprioritize or subdivide requests according to the amount of data to be read/written in each request, but I
Re: swap_pager complaints but not using swap
On Sunday 25 January 2009 04:28:47 Dieter wrote: AMD64 FreeBSD 7.0 2 GiB main memory So the machine doesn't normally use swap much at all, but messing with the large ISO apparently kicked something out of memory, and the disk with the swap partition was already busy writing at the other end of the disk. Do you consider writing a large file to disk a truly unspeakable load? Just curious, since you're running 7.0: SCHED_4BSD or SCHED_ULE? And if you have a chance to change it, does the scenario persist? -- Mel Problem with today's modular software: they start with the modules and never get to the software part. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: swap_pager complaints but not using swap
Dieter wrote: AMD64 FreeBSD 7.0 2 GiB main memory My console says: login: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 pstat -sk Device 1K-blocks UsedAvail Capacity /dev/ad6s10 4590208 96 4590112 0% Wow, using a whole 96K of swap. I don't see any disk related complaints in dmesg. Is this something to worry about? Yes, the system was *trying* to do swap I/O and timing out while doing so. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: swap_pager complaints but not using swap
Kris Kennaway wrote: Dieter wrote: AMD64 FreeBSD 7.0 2 GiB main memory My console says: login: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 pstat -sk Device 1K-blocks UsedAvail Capacity /dev/ad6s10 4590208 96 4590112 0% Wow, using a whole 96K of swap. I don't see any disk related complaints in dmesg. Is this something to worry about? Yes, the system was *trying* to do swap I/O and timing out while doing so. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org isn't swapspace supposed to be on a 'b' partition? Are you using swap on a slice 10? how is that possible when the i386/amd64 BIOS can't see more than 4 primary partitions? Kris, would you mind giving input to this? How can there be a s10, and how can you add swapspace to a device that isn't a partition 'b' nor a file backed swapspace? Those were the only two ways I thought was supported for swap. Dieter, does my questions above sound to be a correct interpretation of your disk setup? thanks! --Tim ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: swap_pager complaints but not using swap
Tim Judd wrote: Kris Kennaway wrote: Dieter wrote: AMD64 FreeBSD 7.0 2 GiB main memory My console says: login: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 pstat -sk Device 1K-blocks UsedAvail Capacity /dev/ad6s10 4590208 96 4590112 0% Wow, using a whole 96K of swap. I don't see any disk related complaints in dmesg. Is this something to worry about? Yes, the system was *trying* to do swap I/O and timing out while doing so. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org isn't swapspace supposed to be on a 'b' partition? Are you using swap on a slice 10? how is that possible when the i386/amd64 BIOS can't see more than 4 primary partitions? Kris, would you mind giving input to this? How can there be a s10, and how can you add swapspace to a device that isn't a partition 'b' nor a file backed swapspace? Those were the only two ways I thought was supported for swap. Dieter, does my questions above sound to be a correct interpretation of your disk setup? swap can be put anywhere thesedays (post-FreeBSD 4.x), even on things that are not even simple disk devices. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: swap_pager complaints but not using swap
AMD64 FreeBSD 7.0 2 GiB main memory My console says: login: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 pstat -sk Device 1K-blocks UsedAvail Capacity /dev/ad6s10 4590208 96 4590112 0% Wow, using a whole 96K of swap. I don't see any disk related complaints in dmesg. Is this something to worry about? Yes, the system was *trying* to do swap I/O and timing out while doing so. Kris Whoops, I forgot to change the subject line after adding the k option to pstat. Without the k it said 0 used. And this morning it occurs to me that even if swap used was zero, it could have been trying to *start* using swap. Anyway... given this timeout explaination, I'm guessing that page/swap has to compete with user processes for disk i/o, and thus probably suffers from the same lack of fair i/o scheduling that user processes suffer from. E.g. one process doing disk i/o can lock out another process for at least several minutes, probably indefinitely. :-( ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: swap_pager complaints but not using swap
AMD64 FreeBSD 7.0 2 GiB main memory My console says: login: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22, size: 4096 pstat -sk Device 1K-blocks UsedAvail Capacity /dev/ad6s10 4590208 96 4590112 0% Wow, using a whole 96K of swap. I don't see any disk related complaints in dmesg. Is this something to worry about? Yes, the system was *trying* to do swap I/O and timing out while doing so. isn't swapspace supposed to be on a 'b' partition? Are you using swap on a slice 10? how is that possible when the i386/amd64 BIOS can't see more than 4 primary partitions? Kris, would you mind giving input to this? How can there be a s10, and how can you add swapspace to a device that isn't a partition 'b' nor a file backed swapspace? Those were the only two ways I thought was supported for swap. Dieter, does my questions above sound to be a correct interpretation of your disk setup? Traditionally swap used the b partition. But then traditionally, there weren't MBR style partitions, called slices in FreeBSD-land. I suspect that the computers Unix grew up on (PDP-7, PDP-11, VAX) had to boot from the beginning of the disk, so the a partition went there. The Alpha continues in this DEC tradition. I was about to say that swap went next for speed, since the machines back then never had enough main memory, but those old disks didn't have variable number of sectors on inner vs outer tracks, so the speed would have been the same across the platter. So I'm not sure why swap was next. This machine has 2 GiB of main memory and almost never uses the swap partition, so I put swap at the slow end of the drive. Yes I have swap on slice 10. I use NetBSD's fdisk, as it handles more than 4 slices nicely, unlike FreeBSD's fdisk. As far as I know, the BIOS firmware doesn't need to know about swap. I think the BIOS firmware just loads and runs the MBR, which in turn loads and runs the bootstrap in the selected slice (or loads and runs the MBR in a different disk if you want). I suppose I could put a BSD disklabel on slice 10 and set it up with the whole slice as the b partition. But as far as I can tell FreeBSD is happy with /dev/ad6s10. As I wrote in my previous message I suspect that the pager/swaper is competing for disk i/o. I forgot to ask if there is some sysctl or other knob to lengthen the timeout. The real fix is to improve the i/o fairness, but I've been asking about this for 2-3 years and not getting anywhere. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
