Re: {a}sync updates (was Re: make install trick)
I have noticed similarly odd behaviour from softupdates during heavy IO load, where something is creating lots of little files or directories and not much else is happening. Using `vmstat 1` I can see that softupdates isn't very good at evening out the IO rate over time: there's a roughly sinusoidal back-and-forth between frantic disk thrashing (lots of TPS) and lots of syscall activity. Visible progress is correspondingly uneven. For example, the `vmstat 1` output while the script for i in `jot 256` do for j in `jot 256` do mkdir -p $i/$j done done is running is appended below. Behaviour is much smoother during an `rm -r *` of the resulting directory tree. Tony. -- f.a.n.finch[EMAIL PROTECTED][EMAIL PROTECTED] Apache Software Foundation Member procs memory page disksfaults cpu r b w avm fre flt re pi po fr sr da0 fd0 pa0 in sy cs us sy id 2 1 0 4183836 11172 3389 0 0 0 3188 0 273 0 0 512 6015 1210 71 29 0 1 1 0 4183836 11172 447 0 0 0 444 0 427 0 0 667 3198 1128 93 7 0 1 1 0 4183836 11172 411 0 0 0 408 0 459 0 0 699 3424 1039 89 11 0 1 1 0 4183836 11172 428 0 0 0 425 0 469 0 0 713 3430 1039 96 4 0 2 0 0 4183832 2 3544 0 0 0 3285 0 296 0 0 535 6820 990 67 32 1 3 0 0 4184036 11084 10155 0 0 0 9427 0 60 0 0 298 14837 1026 45 55 0 2 0 0 4184044 11092 7856 0 0 0 7292 0 0 0 0 242 12077 1013 52 48 0 3 0 0 4183588 11216 13700 0 0 0 12718 0 0 0 0 239 18826 960 26 74 0 3 0 0 4184056 6 12824 0 0 0 11880 0 5 0 0 245 17635 977 35 65 0 2 0 0 4183836 11164 8604 0 0 0 8011 0 1 0 0 242 12450 1008 55 45 0 2 1 0 4183900 11168 5770 0 0 0 5386 0 189 0 0 428 9073 1241 57 43 0 1 1 0 4183836 11168 497 0 0 0 489 0 472 0 0 712 3006 1107 91 9 0 1 2 0 4183496 11136 502 0 0 0 483 0 577 0 0 829 3449 1059 88 12 0 1 1 0 4183496 11156 1444 0 0 0 1365 0 538 0 0 779 4620 1049 86 14 0 3 0 0 4183248 11120 11937 0 0 0 11068 0 107 0 0 344 16580 892 22 78 0 2 0 0 4183308 11132 15356 0 0 0 14213 0 1 0 0 241 20742 979 18 82 0 3 0 0 4183704 11048 12665 0 0 0 11713 0 0 0 0 239 17461 990 28 72 0 2 0 0 4183492 11092 10878 0 0 0 10126 0 21 0 0 259 15238 986 45 55 0 2 0 0 4183496 2 12151 0 0 0 11276 0 50 0 0 289 16767 982 28 72 0 2 1 0 4183496 11144 422 0 0 0 425 0 510 0 0 756 2444 1217 84 16 0 1 1 0 4183496 11100 475 0 0 0 462 0 575 0 0 820 3299 1131 87 13 0 1 1 0 4183496 11092 1024 0 0 0 976 0 550 0 0 792 4130 1049 88 12 0 3 0 0 4183648 11104 7254 0 0 0 6747 0 291 0 0 530 11196 949 53 47 0 2 1 0 4183768 11140 12570 0 0 0 11678 0 5 0 0 249 17381 1002 33 67 0 2 0 0 4183832 11028 13877 0 0 0 12846 0 7 0 0 249 18952 986 25 75 0 2 0 0 4183648 11028 12263 0 0 0 11400 0 16 0 0 259 16964 1015 38 62 0 3 0 0 4183768 11036 11108 0 0 0 10353 0 5 0 0 244 15567 1017 32 68 0 3 0 0 4183836 10952 5932 0 0 0 5539 0 190 0 0 429 9087 1153 55 45 0 2 1 0 4183836 10900 475 0 0 0 459 0 542 0 0 782 2865 1257 91 9 0 1 2 0 4183768 10936 766 0 0 0 749 0 562 0 0 809 3732 1219 87 13 0 1 1 0 4184312 10900 853 0 0 0 803 0 557 0 0 799 4189 1142 90 10 0 2 0 0 4184124 10812 11847 0 0 0 10971 0 101 0 0 340 16642 969 29 71 0 2 0 0 4184520 10720 8599 0 0 0 7976 0 6 0 0 244 13098 1065 48 52 0 2 0 0 4184312 10732 7968 0 0 0 7410 0 5 0 0 242 12228 1034 50 49 1 To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
"David Schwartz" wrote: We're talking about the special case of small root partitions, such that softupdates inability to make empty space available quickly can make the difference between a major operation's success or failure. This is almost impossible on a 1.8Gb root partition. [sorry, cannot resist...] real-life story Once upon a time, a month or so ago, there were ~30G of free space on our 130G filesystem (with softupdates). An important application that was going to create a ~35G file was running. It already written out 1G or so. My colleague called me and sayed: "I removed a 10G of files TWO HOURS ago and the space didn't free up yet!!! The free space isn't going to appear!!! What do we do now???" Well, after I stopped the application with kill -STOP, and temporary killed off another I/O consuming program, the free space started to appear and after several minutes there were 40G of free space. /real-life story I think that the problem in this particular case was inability of the syncer to run as fast as it supposed to. It is assummed that syncer fsync 1/30 of all files and process the softupdates worklist every second. If there are several I/O bound processes running, the syncer will not have enough I/O bandwidth to do this job in the required speed. Perhaps running several syncer processes could help. (OTOH, the machine in question is running a quite old version of FreeBSD-CURRENT, it is possible things are already better. I don't see serious changes in softupdates code from that moment, tough. It is also possible that the machine is mistuned in some way, but I don't know how :-() Dima To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
On Thu, Oct 07, 1999 at 03:15:03PM -0700, David Schwartz wrote:
There should be fairly few writes to the root partition, so having
An opionion. I use the HP workstation model where my / is 1800M. I have
You are not disagreeing with him, David. You are just talking about
another scenario other than the one under discussion.
He was talking about the case where root is small. This whole discussion
was about how softupdates behaves in the subcase of small root partitions.
This discussion was NOT about "how softupdates behaves in the subcase of
small root partitions" Imp was having problems in the face of
softupdates on a full /. The problem exists *reguardless* of how big /
is, the issues is % free.
If you have a 1.8Gb root partition that also includes /var and /usr, this
whole discussion is irrelevant.
Why?? Because / can now not fill up? I've installed enough
KDE/GNOME/teTeX/etc... ports (plus /usr/{ports,src}) that I have acutally
filled it up before.
(And before I'm attacked for my organization -- if I can get it back from
the CDROM and it lives in /usr, it is on the / file systems which I don't
back up, as there is no use to.)
--
-- David([EMAIL PROTECTED])
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
RE: {a}sync updates (was Re: make install trick)
On Thu, Oct 07, 1999 at 03:15:03PM -0700, David Schwartz wrote:
There should be fairly few writes to the root partition, so having
An opionion. I use the HP workstation model where my / is
1800M. I have
You are not disagreeing with him, David. You are just talking about
another scenario other than the one under discussion.
He was talking about the case where root is small. This
whole discussion
was about how softupdates behaves in the subcase of small
root partitions.
This discussion was NOT about "how softupdates behaves in the subcase of
small root partitions" Imp was having problems in the face of
softupdates on a full /. The problem exists *reguardless* of how big /
is, the issues is % free.
No, this is a different issue. The problem was not that the filesystem was
full, but that the fill rate was comparable to the amount of 'empty' space.
If you have a 1.8Gb root partition that also includes /var
and /usr, this
whole discussion is irrelevant.
Why?? Because / can now not fill up? I've installed enough
KDE/GNOME/teTeX/etc... ports (plus /usr/{ports,src}) that I have acutally
filled it up before.
Yes, but if you fill up your root partition, there's nothing that can be
done. Full is full.
We're talking about the special case of small root partitions, such that
softupdates inability to make empty space available quickly can make the
difference between a major operation's success or failure.
This is almost impossible on a 1.8Gb root partition. And if you had a 1.8Gb
partition with only 15Mb free, the last thing you'd care about is how
softupdates will handle the situation.
DS
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
We're talking about the special case of small root partitions, such that softupdates inability to make empty space available quickly can make the difference between a major operation's success or failure. This is almost impossible on a 1.8Gb root partition. Again why? What's the difference between a small / and a 1.8GB (byte not bit) one? There is nothing here to back this up. Not enough space is not enough space reguardless how much you started out with. And if you had a 1.8Gb partition with only 15Mb free, the last thing you'd care about is how softupdates will handle the situation. Why would I be so concerned? If I don't expect to need that 15M then, I've sized my partition just right. Don't put cares in my basket that don't need to be there. More uninstatiated opions. -- -- David([EMAIL PROTECTED]) To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
On Thu, Oct 07, 1999 at 11:57:26AM +1000, a little birdie told me that Peter Jeremy remarked How detailed should the man page be? Exactly my query in writing this ; If it stated "all file data will be written synchronously, but inodes where the only update is atime and free block bitmaps are written asynchronously", would that be any clearer to a user who didn't have a detailed understanding of UFS? If you would like it to say something different, write some patches and send them in as a PR (keeping in mind phk's recent e-mail about green bikesheds). I'm still stewing on what should be with it; what we have works fine, if being slightly inconsistent in an obscure way. I'm trolling for ideas on whether well enough should be left alone (since there's obviously an incredibly small percentage of people USING sync as a mountop), or whether a footnote should be added somewhere (I lean toward mount(2) instead of mount(8) myself, with a possible xref in mount(8)). I'll see what I think of, and possibly have some diffs tomorrow. There should be fairly few writes to the root partition, so having these writes synchronous is not a big performance hit. On the other hand, there are probably a _lot_ of read accesses to devices in /dev and files in /bin (how many of your scripts begin #!/bin/sh?). Unless you specify NOATIME, each of these read accesses implies an atime update within the inode. Making these synchronous probably would be a big performance hit. This is why I haven't screamed for them to be sync-tified... -- Matthew Fuller (MF4839) |[EMAIL PROTECTED] Unix Systems Administrator |[EMAIL PROTECTED] Specializing in FreeBSD |http://www.over-yonder.net/ FutureSouth Communications |ISPHelp ISP Consulting "The only reason I'm burning my candle at both ends, is because I haven't figured out how to light the middle yet" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
Maybe the best solution is the following: - leave "sync" with its current behaviour - create a sysctl to make it truely synchronous (I was thinking of a new mount option but thats overkill) and have the documentation for that sysctl state the performance hit and recommend that the filesystem be mounted with "noatime" when this sysctl is on. The sysctl could have three levels: - off - on for atime updates - on for atime updates and free block bitmap updates On Thu, 7 Oct 1999, Peter Jeremy wrote: On 1999-Oct-07 09:15:42 +1000, Matthew D. Fuller wrote: Is this good, bad, ugly, or just inconsistent? On the one hand, you can argue that 'sync should be sync should be sync, I don't bloody care, just don't do anything async at all', since that's what it's supposed to do: mount(8): syncAll I/O to the file system should be done synchronously. How detailed should the man page be? If it stated "all file data will be written synchronously, but inodes where the only update is atime and free block bitmaps are written asynchronously", would that be any clearer to a user who didn't have a detailed understanding of UFS? If you would like it to say something different, write some patches and send them in as a PR (keeping in mind phk's recent e-mail about green bikesheds). sync atime updates will slow it down, but on the flip side, if you're mounting sync in the first place you don't care much for speed anyway. There should be fairly few writes to the root partition, so having these writes synchronous is not a big performance hit. On the other hand, there are probably a _lot_ of read accesses to devices in /dev and files in /bin (how many of your scripts begin #!/bin/sh?). Unless you specify NOATIME, each of these read accesses implies an atime update within the inode. Making these synchronous probably would be a big performance hit. Peter -- /===\ | Work: [EMAIL PROTECTED] | Home: [EMAIL PROTECTED] | \===/ "If it is true that our Universe has a zero net value for all conserved quantities, then it may simply be a fluctuation of the vacuum of some larger space in which our Universe is imbedded. In answer to the question of why it happened, I offer the modest proposal that our Universe is simply one of those things which happen from time to time." E. P. Tryon from "Nature" Vol.246 Dec.14, 1973 To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
There should be fairly few writes to the root partition, so having An opionion. I use the HP workstation model where my / is 1800M. I have no use for /var and /usr and find them simply stupid in today's world. (except for ISP's where there is cause for a septerate /var). Lets stick to facts. -- -- David([EMAIL PROTECTED]) To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
mount(8): syncAll I/O to the file system should be done synchronously. How detailed should the man page be? If it stated "all file data will be written synchronously, but inodes where the only update is atime and free block bitmaps are written asynchronously", would that be any clearer to a user who didn't have a detailed understanding of UFS? Yes. I know the difference between sync/async and data/metadata. I haven't however, done a though wall-thru of the UFS code. If the manpage says "All I/O", then the system should do *ALL* I/O sync. This isn't Linux. Guess I go off editing mount(8) again. -- -- David([EMAIL PROTECTED]) To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
RE: {a}sync updates (was Re: make install trick)
There should be fairly few writes to the root partition, so having An opionion. I use the HP workstation model where my / is 1800M. I have no use for /var and /usr and find them simply stupid in today's world. (except for ISP's where there is cause for a septerate /var). Lets stick to facts. -- -- David([EMAIL PROTECTED]) You are not disagreeing with him, David. You are just talking about another scenario other than the one under discussion. He was talking about the case where root is small. This whole discussion was about how softupdates behaves in the subcase of small root partitions. If you have a 1.8Gb root partition that also includes /var and /usr, this whole discussion is irrelevant. DS To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
On 1999-Oct-08 08:13:12 +1000, David O'Brien wrote: mount(8): syncAll I/O to the file system should be done synchronously. How detailed should the man page be? If it stated "all file data will be written synchronously, but inodes where the only update is atime and free block bitmaps are written asynchronously", would that be any clearer to a user who didn't have a detailed understanding of UFS? Yes. I know the difference between sync/async and data/metadata. My point was that the average user probably doesn't. I agree that the current description glosses over the difference (and probably shouldn't). We need to strike a balance between providing enough detail for the knowledgeable user (who wants to know exactly what a sync mount does to different types of writes within the FS) and the novice user (who doesn't understand the details of UFS and is more likely to become confused). IMHO, `sync' does behave in a reasonable manner. I'm not sure how I'd go about explaining its behaviour to someone who didn't understand the UFS though. Guess I go off editing mount(8) again. I was going to suggest that the relationship between the sync option in mount(2,8) and O_FSYNC in open(2) be noted. Only problem is that O_FSYNC isn't documented :-(. Peter -- Peter Jeremy (VK2PJ)[EMAIL PROTECTED] Alcatel Australia Limited 41 Mandible St Phone: +61 2 9690 5019 ALEXANDRIA NSW 2015 Fax: +61 2 9690 5982 To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
:mount(8): : syncAll I/O to the file system should be done synchronously. : :On the gripping hand, you can say, 'this is an ATIME update, there's no :way its presence or lack thereof can do anything bad to the filesystem, :so let it be async since it takes extra work to make it sync'. : :Does anyone have any feeling either way on this? I, unfortunately, seem :to have strong feelings BOTH ways... sync atime updates will slow it :down, but on the flip side, if you're mounting sync in the first place :you don't care much for speed anyway. : :Thoughts? :Matthew Fuller (MF4839) |[EMAIL PROTECTED] Well, you don't gain anything by making atime updates sync, and you lose a lot, so why do it? If you are worried about protecting the root drive from a crash, mount it read-only (and put /dev on an MFS mount) or mount it noatime. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: {a}sync updates (was Re: make install trick)
On 1999-Oct-07 09:15:42 +1000, Matthew D. Fuller wrote: Is this good, bad, ugly, or just inconsistent? On the one hand, you can argue that 'sync should be sync should be sync, I don't bloody care, just don't do anything async at all', since that's what it's supposed to do: mount(8): syncAll I/O to the file system should be done synchronously. How detailed should the man page be? If it stated "all file data will be written synchronously, but inodes where the only update is atime and free block bitmaps are written asynchronously", would that be any clearer to a user who didn't have a detailed understanding of UFS? If you would like it to say something different, write some patches and send them in as a PR (keeping in mind phk's recent e-mail about green bikesheds). sync atime updates will slow it down, but on the flip side, if you're mounting sync in the first place you don't care much for speed anyway. There should be fairly few writes to the root partition, so having these writes synchronous is not a big performance hit. On the other hand, there are probably a _lot_ of read accesses to devices in /dev and files in /bin (how many of your scripts begin #!/bin/sh?). Unless you specify NOATIME, each of these read accesses implies an atime update within the inode. Making these synchronous probably would be a big performance hit. Peter -- Peter Jeremy (VK2PJ)[EMAIL PROTECTED] Alcatel Australia Limited 41 Mandible St Phone: +61 2 9690 5019 ALEXANDRIA NSW 2015 Fax: +61 2 9690 5982 To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
