Re: gvinum raid5 on 8.0-RELEASE
On Wed, Jun 02, 2010 at 12:58:06PM +0400, venom wrote: Hi, all. I have problem with perfomance on raid5 volume. I have gotten a report from others experiencing this as well. It might be a performance regression, but make sure that your parity is ok: sysctl kern.geom.vinum.debug=4 gvinum checkparity raid5.p0 If you're able to, please apply the following patch in order for the list commands to track your rebuild status with the 'gvinum list -v plex' command: http://people.freebsd.org/~lulf/gvinum_setflag.diff Any errors during checkparity should come out in the console. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: gvinum raid5 on 8.0-RELEASE
On Wed, Jun 02, 2010 at 05:46:24PM +0400, venom wrote: On 06/02/2010 02:13 PM, Ulf Lilleengen wrote: sysctl kern.geom.vinum.debug=4 gvinum checkparity raid5.p0 source is patched, module is rebuilded and reloaded As you can see, the parity is incorrect. Try running 'gvinum rebuildparity raid5.p0'. Jun 2 13:33:19 kernel: GEOM_VINUM[2]: event 'check' Jun 2 13:33:19 kernel: GEOM_VINUM[0]: parity incorrect at offset 0x0 ^^ Jun 2 13:33:19 kernel: GEOM_VINUM[0]: parity check on raid5.p0 failed at 0x0 errno 35 Jun 2 13:33:19 kernel: GEOM_VINUM[2]: event 'drive tasted' Jun 2 13:33:19 kernel: GEOM_VINUM[2]: tasted drive on 'ada1p2' Jun 2 13:33:19 kernel: GEOM_VINUM[2]: drive 'r0' is already known Jun 2 13:33:19 kernel: GEOM_VINUM[2]: event 'drive tasted' Jun 2 13:33:19 kernel: GEOM_VINUM[2]: tasted drive on 'ada2p2' Jun 2 13:33:19 kernel: GEOM_VINUM[2]: drive 'r1' is already known Jun 2 13:33:19 kernel: GEOM_VINUM[2]: event 'drive tasted' Jun 2 13:33:19 kernel: GEOM_VINUM[2]: tasted drive on 'ada3p2' Jun 2 13:33:19 kernel: GEOM_VINUM[2]: drive 'r2' is already known # dd if=/dev/zero of=file count=16 bs=512k 16+0 records in 16+0 records out 8388608 bytes transferred in 38.693080 secs (216799 bytes/sec) ** -- Vladimir Ermakov ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: GEOM_ULZMA
On Fri, Feb 19, 2010 at 04:36:44PM +0200, Alexandr Rybalko wrote: Hi, I wrote a module GEOM_ULZMA (such as GEOM_UZIP, but compression with lzma), in connection with this is an issue best left lzma code in the file geom_ulzma.c or store lzma library separately. If separately, then where better? Maybe in future make lzma and gzip library kernel interface for embedded? Then in one instance of code, userland can use compression via kernel. What are the cons against combining uzip/ulzma into a geom_z/geom_compress module that can support different compression schemes? I think this makes more sense than having different geom modules for each compression scheme. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
[HEADSUP] Google SoC work on ext2fs to be committed
Hello, In a few days, I will commit the ext2fs improvements done by Aditya Sarawgi during and after Google Summer of Code to CURRENT. Some quick notes: - BSD licensed block and inode allocation policies for ext2fs. This involves the use FFS1 style block and inode allocation for ext2fs. Preallocation was removed since it was GPL'd. - Orlov Block Allocator for ext2fs. (Aditya can perhaps provide some details). - Some PR fixes and older bugs. - Make ext2fs MPSAFE by introducing locks to per-mount datastructures. More information along with nice shiny graphs can be found here: http://wiki.freebsd.org/SOC2009AdityaSarawgi and, of course, the code: http://gitorious.org/freebsd-ext2fs/freebsd-ext2fs As the work on support MP-safe operation may cause race conditions not discovered yet, it is important to report on these issues as they are encountered. Issues with the new implementation may be adressed to Aditya (CCed). http://wiki.freebsd.org/AdityaSarawgi -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threads and SMP kernel scheduling
On Thu, Jan 08, 2009 at 04:23:08AM -0500, Mehmet Ali Aksoy TÜYSÜZ wrote: Hi all, After I had a bit googling I got confused. My questions are simple and they are as follows : 1-) Are pthreads (or threads in general) of one process scheduled to different cores on multi-core systems running Linux or BSD? The standard threading library in FreeBSD will use a 1:1 mapping between userland threads(pthreads) and kernel threads. This means that each thread may run on a different core than other threads. 2-) What if there are multiple processes which have multiple threads? Does it change the answer of (1)? No, the same mapping applies. Although threads of one process may preferably run on the same core, each thread of a process may run on any of the cores available. I found some answers but they are not sharp. Somebody says can be scheduled but can be is not a precise answer (in my opinion.) Well, it means that there are more factors deciding where a thread is put. If you look at a factor such as affinity, one thread may preferable be scheduled to the same core since the cache may contain data relevant to the thread. Thanks everybody in advance. Hope everything is at least a bit clearer. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threads and SMP kernel scheduling
On tor, jan 08, 2009 at 09:16:26am -0500, Mehmet Ali Aksoy TÜYSÜZ wrote: Hi, Thank you very much for your response Ulf. It is a very clear answer. Thanks again. By the way, any information for the Linux case? I think this applies to Linux as well, since it's NPTL(Native Posix Threading Library) uses a 1:1 mapping also. I'm not 100% sure, so you might want to ask on a linux mailing list. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
[PATCH] New patch for cvsmode in csup
Hello, Time again for a csup patch. There have been many improvements to make it behave like cvsup in most circumstanses. A few notes: * Fixed a bug where csup would crash if the diff was not applied correctly, requiring changing the procedure on how diffs are applied within csup. * Fix so files added/removed to/from Attic will actually get removed from the client, making it work equally to csup. * Fix so the checkouts/status-file is mostly correctly updated, but this does only matter a little if one mixes usage with csup/cvsup. * Note that updating of CVSROOT-* might take longer time than cvsup, because cvsup supports rsync algorithm, which fits those files better. * A lot of cleanups to the code, making it ready for review. Thanks to kris@ for helping out with testing. If anyone would like to review this patch, that would be nice. The patch can be found here: http://people.freebsd.org/~lulf/patches/csup/csup_09_16_2008.diff -- Ulf Lilleengen pgpLpqXfdx7Qz.pgp Description: PGP signature
[Call for testers] New cvsmode for csup patch
Hello again, Even in these new subversion days, I'm not giving up :) Thanks for the feedback I got on the previous patches. It brought some issues with the patch that should now be fixed. It took some time, but I've been priotitizing other projects. However, here are a list of the changes: - Someone experienced segfaults when doing an update. This was because using wrong format specifier to proto_printf. - Some parts of the protocol was missing, such as support for the Z command, and supporting U in cvsmode. This is now implemented. - The rcsfile implementation would use huge amounts of memory when running diffs. Lately, it was easier to pinpoint where this happened due to huge diff chains, showing clearly where the memory was lost. This is now fixed, and csup seems to have a stable memory usage. Please report if you see otherwise. - Some simplifications and removal of code that is not needed. - Added an update of the usr.bin/csup Makefile. Thanks to kris@ for pointing this out. I hope all of you that tried the previous patch will try this one. Your previous feedback was valuable in finding out where I should look to fix these problems. Patches can be found for CURRENT: http://people.freebsd.org/~lulf/patches/csup/csup_2008-08-01_CURRENT.diff and 7-STABLE: http://people.freebsd.org/~lulf/patches/csup/csup_2008-08-01_RELENG_7.diff Any feedback welcome! -- Ulf Lilleengen pgpO7LhB34uZD.pgp Description: PGP signature
Re: CFT: CVSMode for csup with MD5 check
On lør, jun 07, 2008 at 10:18:05am +0200, Kris Kennaway wrote: Ulf Lilleengen wrote: Hello, As a followup to my previous patch on csup, I've tried to do some fixes to RCS-files. However, instead of doing major workarounds in csup to handle files which does not behave correctly to RCS, I implemented MD5 check of RCS content. This means that the MD5 sum from cvsupd is checked against the actual RCS content (which is different from a normal MD5 check, because only the content is compared), and if it's not correct, a fixup of the file will be sent, thus making sure that the contents are correct. I hope some of you are able to test this. There are still a few issues with cvsmode: - Not correct entries in status file. - I get unnatural high memory usage. Patches here: http://people.freebsd.org/~lulf/patches/csup/csup_2008-05-31_CURRENT.diff and http://people.freebsd.org/~lulf/patches/csup/csup_2008-05-31_RELENG_7.diff This doesn't compile. There are a bunch of warnings, and rcsfile.c is missing from Makefile, but there is an undefined reference: rcsfile.o(.text+0x1d32): In function `rcsfile_frompath': : undefined reference to `rcsparse_run' *** Error code 1 Sorry, I probably didn't test the patch correctly. These patches now works for me on a 7.0 and CURRENT tree at least: http://people.freebsd.org/~lulf/patches/csup/csup_2008-06-07_RELENG_7.diff http://people.freebsd.org/~lulf/patches/csup/csup_2008-06-07_CURRENT.diff -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
CFT: CVSMode for csup with MD5 check
Hello, As a followup to my previous patch on csup, I've tried to do some fixes to RCS-files. However, instead of doing major workarounds in csup to handle files which does not behave correctly to RCS, I implemented MD5 check of RCS content. This means that the MD5 sum from cvsupd is checked against the actual RCS content (which is different from a normal MD5 check, because only the content is compared), and if it's not correct, a fixup of the file will be sent, thus making sure that the contents are correct. I hope some of you are able to test this. There are still a few issues with cvsmode: - Not correct entries in status file. - I get unnatural high memory usage. Patches here: http://people.freebsd.org/~lulf/patches/csup/csup_2008-05-31_CURRENT.diff and http://people.freebsd.org/~lulf/patches/csup/csup_2008-05-31_RELENG_7.diff -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Call for testers: CVSMode for csup
Hello all, During the past few months, I've implemented CVSMode support for csup. This means one can use csup to fetch complete CVS repositories. However, first I'd like everyone who'd like to help, to try out this patch and test to find bugs and issues with it. Currently, I'm pretty sure it should behave correctly in the normal cases, but therefore needs to be tested by a wider audience. Also, there are some flaws that are noted at the bottom of this e-mail, but the important thing is to test the correctness regarding RCS. For now, I'm including the tokenizer generated by flex since the base system flex won't be good enough yet. Hopefully, it'll get updated soon. A more technical overview: The support for CVSMode is accomplished by extending the already good foundation of csup to support the commands listed in the cvsup protocol. This means that I've added support for this in the detailer (the part giving the cvsup server the client file info) and updater(the one taking orders from the server) part of csup. In addition, I've created the rcsfile-interface. The interface specifies functions to add and remove deltas, tags, attributes etc. of a RCS file. To initially read a RCS file, I've created a parser using flex as tokenizer. Since the parser and tokenizer must be reentrant (both updater and detailer uses it), a fairly new version of flex is needed. There are some known issues with this patch: - Some RCS files contains extra whitespaces due to hackery. CVSup solves this by counting them during reading. This could perhaps be solved by using a newline counter in the lexxer, but I'm not sure if this is really necessary. - Some RCSfiles such as src/share/examples/kld/firmware/fwimage/firmware.img,v differ because there are some unknown garbage I think, but this should be investigated and confirmed. - It has a quite high memory usage, and this might be due to some leaks that I've been unable to find. I'll do a much better audit of the code and run valgrind to investigate this further. - Does not support md5 of RCS stream, so it can't detect errors yet. - Statusfile file attributes might not be correct. - Some RCS parts such as newphrases (man rcsfile) is not supported yet. - Some hardcoded limits that may break it. - Things done a silly way such as sorting and comparing, which I have plans to improve later. Please consider this as a very pre-alpha patch :) Given this list, I'm not so sure unleashing this patch is so good after all :), but it's really very experimental for now. As i mentioned, the things I want to be tested right now is if it does the RCS update procedure correctly in all cases (except the points I just mentioned). I've been testing on some parts of the FreeBSD src repository (diffing the result against cvsup result), and this is a quite good testcase, but different repos is always good. The latest patches for CURRENT and RELENG_7 can be found here: http://people.freebsd.org/~lulf/patches/csup/cvsmode Also, I'll be on and off the internet during the next few weeks (in Japan until 25th of march), so I might respond a bit late. I'll come back with more improvements and new patches as soon as I can. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Call for testers: CVSMode for csup
On Thu, Mar 06, 2008 at 08:22:24AM -0800, David O'Brien wrote: On Thu, Mar 06, 2008 at 01:44:59PM +0100, Ulf Lilleengen wrote: During the past few months, I've implemented CVSMode support for csup. This would be good to have. Have you looked at CVSync http://www.cvsync.org/ to see how he handled some of the wierdness in ,v files? No, but I've looked at cvsup. Most of the wierdness is handled, only two issues listed in my e-mail remains, and they should not be too hard to fix. However, I'm not sure how important they are either, since it's actually a matter of different amounts of newlines which doesn't have anything to do with the RCS file structure itself. But thanks for the tip anyway. I'll take a look at cvssync. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Csup cvsmode build discussion
On ons, jan 16, 2008 at 04:15:53pm +0100, Philip Paeps wrote: On 2008-01-16 16:06:56 (+0100), Ulf Lilleengen [EMAIL PROTECTED] wrote: However, this might only fix the issue with yacc if I'm thinking correctly, because the lexer also needs to be told about this for reentrancy. (But perhaps a much smaller problem since it's a matter of updating the lex version in base). I found that updating the lex in base was surprisingly easy. :-) I'll clean up my patches and post them for review later this evening. Excellent, thank you for your work on this Philip. I also found a good way to do what I planned. I'll use the updated flex in base, and then write the parser in C. This shouldn't be too hard since I have a tokenizer and just need to assert the correct structure of the file. This way I won't have to deal with yacc at all, which is sort of a relief :) -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Csup cvsmode build discussion
Hello hackers, As some might know, I've been implementing CVSMode for csup over the last couple of months. Right now, I'm quite close to have it work (does not mean it will work perfect :)), but I encountered some issues regarding the inclusion of my lex and yacc files that I would like to discuss. It is assumed that the reader know what lex and yacc is. First of all, I implemented the RCS parser required to operate on RCS files using lex and yacc. Those are in FreeBSD userland, but the problem is they are wayy to outdated for csups need. The reason is mainly because they don't support: 1) reentrancy (needed since csup uses two threads, one to send information to the server regarding it's files, and one to recieve information on how a file is to be updated). 2) prefixing of lex/yacc output. Not a big deal, but easy if one doesn't need to have to work around it. 3) parameter passing. Part of the reentrancy requirement but useful if one would extend the arguments to a parser. Because of this, I've been compiling the parser and tokenizer with the help of bison and flex from ports. Now, the base system already have flex, but the flex version in base is heavily outdated (version 2.5.4 versus 2.5.34 in ports) and does not support reentrancy. Is there a reason why it's outdated? What I can think of is that it is either unmaintained, or that developers want it to disappear from base. The yacc version in base is pretty old too and does not support reentrancy. I would like to not have to hack it up to support this to avoid using bison, (And I suspect importing bison into base is going to happen over someones dead body :)). So, I'm asking you, how should I best deal with this? Should I a) Just include the _generated_ parser and tokenizer (e.g the output from bison and flex) and just include the yacc and lex files in case someone wants to work more on it? b) Modify parser to be non-reentrant and solve my issues with locks. c) Import bison into base and update flex version. d) Insert your fantastic idea here Alternative 'a' is my favorite, and seems to be the easiest way to avoid a heavy inpact on the base system, but again it will complicate procedure of modifying the parser since one have to use the correct build tools etc. 'b' would lead to performance loss and I would very much like to avoid it. 'c' would take a lot of work and I think many would protest on importing bison (including me if I were not biased with needing it and all :)) What I ask of you is comments on these alternatives as well as inserting your fantastic idea if you happen to have one :) -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Csup cvsmode build discussion
On Wed, Jan 16, 2008 at 11:08:20AM +, Bob Bishop wrote: Hi, On 16 Jan 2008, at 09:33, Ulf Lilleengen wrote: Hello hackers, As some might know, I've been implementing CVSMode for csup over the last couple of months. Right now, I'm quite close to have it work (does not mean it [...] What I ask of you is comments on these alternatives as well as inserting your fantastic idea if you happen to have one :) I'm not going to comment on the merits of flex and bison as I like to be polite :-) I'd suggest (a), but package the yacc and lex source files up as a port so that they are in CVS and can readily be rebuilt if necessary. Perhaps. Not very pretty. Why do you want prefixing? And precisely what do you want to do with parameters? Prefixing is needed since csup already have a configuration parser written in lex/yacc and therefore needed to avoid conflicts. Parameters is nice if one were to modify the parser to take extra arguments, I don't have a concrete example for it yet, but it's nonetheless a requirement for reentrancy. An alternative d) that I got from Maxime is that I'll take a look and see if I can use a parser from cvsync or from the previous SoC 2005 work, which I think is a good idea. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Csup cvsmode build discussion
On Wed, Jan 16, 2008 at 07:41:47AM -0600, Matthew D. Fuller wrote: On Wed, Jan 16, 2008 at 02:32:14PM +0100 I heard the voice of Ulf Lilleengen, and lo! it spake thus: On Wed, Jan 16, 2008 at 11:08:20AM +, Bob Bishop wrote: Why do you want prefixing? Prefixing is needed since csup already have a configuration parser written in lex/yacc and therefore needed to avoid conflicts. Does lex's -P and yacc's -p not rewrite enough? This was just pointed out to me. Sorry for the bad research. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Csup cvsmode build discussion
On ons, jan 16, 2008 at 02:39:26pm +, Bob Bishop wrote: Hi, On 16 Jan 2008, at 13:32, Ulf Lilleengen wrote: On Wed, Jan 16, 2008 at 11:08:20AM +, Bob Bishop wrote: [...] Why do you want prefixing? And precisely what do you want to do with parameters? Prefixing is needed since csup already have a configuration parser written in lex/yacc and therefore needed to avoid conflicts. You can alternatively avoid conflicts by putting both grammars in the same file, with an entry rule which refers to one or other of the two grammars' real entries based on a defined token. You then arrange for the lexer to supply the appropriate value of the switch token each time the parser is started up. As noted by others in this thread, the lex and yacc versions in base does in fact support prefixing. Thus, the only problem is reentrancy, which I've commented on below. Parameters is nice if one were to modify the parser to take extra arguments, I don't have a concrete example for it yet, but it's nonetheless a requirement for reentrancy. [etc] You may be able to do this using a variation on the above trick. So for the reentrancy part, what you suggest is that instead of passing the argument to the reentrant parser, I create the global variables for each thread using the parser, and then use one or the other global variables depending on which thread is calling the parser? It's a very interesting trick, and I think it might work. It's not very scalable, but it's a good workaround to be able to use the base system lex and yacc. However, this might only fix the issue with yacc if I'm thinking correctly, because the lexer also needs to be told about this for reentrancy. (But perhaps a much smaller problem since it's a matter of updating the lex version in base). Anyway, thanks for a useful suggestion. I'll try it out. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Patch RFC: Promise SATA300 TX4 hardware bug workaround.
On man, nov 19, 2007 at 09:02:33 +0100, Søren Schmidt wrote: Hi All! I'd like to get the final verdict of the attached patch and if it fixes the problem or not. Please test and report, its a bit urgent if it need to get into R7 :) Hi! I'm sorry I wasn't able to test this earlier, but my office was locked during the weekend and I was therefore not able to test until today. But good news is, it works. I get no error messages when reading or writing data to the drives anymore, and the partition table is correctly read so that the correct device nodes show up. This should definately go into 7.0 imho if no bugs show up. Thanks! -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Patch RFC: Promise SATA300 TX4 hardware bug workaround.
On fre, nov 02, 2007 at 01:34:51 +0300, Alexander Sabourenkov wrote: Hello. I have ported the workaround for the hardware bug that causes data corruption on Promise SATA300 TX4 cards to RELENG_7. Bug description: SATA300 TX4 hardware chokes if last PRD entry (in a dma transfer) is larger than 164 bytes. This was found while analysing vendor-supplied linux driver. Workaround: Split trailing PRD entry if it's larger that 164 bytes. Two supplied patches do fix problem on my machine. There is, however, a style problem with them. It seems like PRD entry count is limited at 256. I have not found a good way to guarantee that one entry is always available to do the split, thus the ugly solution of patching ata-dma.c. Patches, patched and original files are at http://lxnt.info/tx4/freebsd/. Hi, I tried the patch, but I end up with the partition table being incorrectly read (probably) on the drives connected to my TX4 card. Normally, there's one partition on the drive, but when I apply the patch, the drive provider (ad6) is all that shows up in /dev. When I revert the patch, the partition (ad6s1) shows up in /dev again. I applied both the ata-chipset patch and ata-dma patch to a RELENG_7 system. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Pluggable Disk Scheduler Project
On tir, okt 16, 2007 at 04:10:37 +0200, Karsten Behrmann wrote: Hi, is anybody working on the `Pluggable Disk Scheduler Project' from the ideas page? I've been kicking the idea around in my head, but I'm probably newer to everything involved than you are, so feel free to pick it up. If you want, we can toss some ideas and code to each other, though I don't really have anything on the latter. [...] After reading [1], [2] and its follow-ups the main problems that need to be addressed seem to be: o is working on disk scheduling worth at all? Probably, one of the main applications would be to make the background fsck a little more well-behaved. I agree, as I said before, the ability to give I/O priorities is probably one of the most important things. o Where is the right place (in GEOM) for a disk scheduler? [...] o How can anticipation be introduced into the GEOM framework? I wouldn't focus on just anticipation, but also other types of schedulers (I/O scheduling influenced by nice value?) o What can be an interface for disk schedulers? good question, but geom seems a good start ;) o How to deal with devices that handle multiple request per time? Bad news first: this is most disks out there, in a way ;) SCSI has tagged queuing, ATA has native command queing or whatever the ata people came up over their morning coffee today. I'll mention a bit more about this further down. o How to deal with metadata requests and other VFS issues? Like any other disk request, though for priority-respecting schedulers this may get rather interesting. [...] The main idea is to allow the scheduler to enqueue the requests having only one (other small fixed numbers can be better on some hardware) outstanding request and to pass new requests to its provider only after the service of the previous one ended. [...] - servers where anticipatory performs better than elevator - realtime environments that need a scheduler fitting their needs - the background fsck, if someone implements a priority scheduler Apache is actally a good candidate according to the old antipacitory design document ( not sure of it's relevance today, but...) Over to a more general view of it's architecture: When I looked at this project for the first time, I was under the impression that this would be best done in a GEOM class. However, I think the approach that was taken in the Hybrid project is better because of the following reasons: - It makes it possible to use by _both_ GEOM classes and device drivers (Which might use some other scheduler type?). - Does not remove any configuratbility, since changing etc. can be done by user with sysctl. - Could make it possible for a GEOM class to decide for itself which scheduler it wants to use (most GEOM classes uses the standard bioq_disksort interface in disk_subr.c). - The ability to stack a GEOM class with a scheduler could easily be emulated by creating a GEOM class to utilize the disksched framework. All in all, I just think this approach gives more flexibility than putting it in a GEOM class that have to be added manually by a user. Just my thought on this. Also, I got my test-box up again today, and will be trying your patch as soon as I've upgraded it to CURRENT Fabio. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Pluggable Disk Scheduler Project
On ons, okt 17, 2007 at 02:19:07 +0200, Fabio Checconi wrote: From: Ulf Lilleengen [EMAIL PROTECTED] Date: Wed, Oct 17, 2007 01:07:15PM +0200 On tir, okt 16, 2007 at 04:10:37 +0200, Karsten Behrmann wrote: Over to a more general view of it's architecture: When I looked at this project for the first time, I was under the impression that this would be best done in a GEOM class. However, I think the approach that was taken in the Hybrid project is better Ok. I think that such a solution requires a lot more effort on the design and coding sides, as it requires the modification of the drivers and can bring us problems with locking and with the queueing assumptions that may vary on a per-driver basis. I completely agree with the issue of converting device drivers, but at least it will be an _optional_ possibility (Having different scheduler plugins could make this possible). One does not necessary need to convert the drivers. Maybe I've not enough experience/knowledge of the driver subsystem, but I would not remove the queueing that is done now by the drivers (think of ata freezepoints,) but instead I'd like to try to grab the requests before they get to the driver (e.g., in/before their d_strategy call) and have some sort of pull mechanism when requests complete (still don't have any (serious) idea on that, I fear that the right place to do that, for locking issues and so on, can be driver dependent.) Any ideas on that? Which drivers can be good starting points to try to write down some code? If you look at it, Hybrid is just a generalization of the existing bioq_* API already defined. And this API is used by GEOM classes _before_ device drivers get the requests AFAIK. For a simple example on a driver, the md-driver might be a good place to look. Note that I have little experience and knowledge of the driver subsystem myself. Also note (from the Hybrid page): * we could not provide support for non work-conserving schedulers, due to a couple of reasons: 1. the assumption, in some drivers, that bioq_disksort() will make requests immediately available (so a subsequent bioq_first() will not return NULL). 2. the fact that there is no bioq_lock()/bioq_unlock(), so the scheduler does not have a safe way to generate requests for a given queue. This certainly argues for having this in the GEOM layer, but perhaps it's possible to change the assumtions done in some drivers? The locking issue should perhaps be better planned though, and an audit of the driver disksort code is necessary. Also: * as said, the ATA driver in 6.x/7.x moves the disksort one layer below the one we are working at, so this particular work won't help on ATA-based 6.x machines. We should figure out how to address this, because the work done at that layer is mostly a replica of the bioq_*() API. So, I see this can get a bit messy thinking of that the ATA drivers does disksorts on its own, but perhaps it would be possible to fix this by letting changing the general ATA driver to use it's own pluggable scheduler. Anyway, I shouldn't demand that you do this, especially since I don't have any code or anything to show to, and because you decide what you want to do. However, I'd hate to see the Hybrid effort go to waste :) I was hoping some of the authors of the project would reply with their thoughts, so I CC'ed them. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Pluggable Disk Scheduler Project
ago, so it should be tried again though. [1] http://wiki.freebsd.org/Hybrid [2] http://lists.freebsd.org/pipermail/freebsd-geom/2007-January/001854.html [3] The details of the anticipation are really not interesting as it is extremely simplified by purpose. [4] http://feanor.sssup.it/~fabio/freebsd/g_sched/ contains also an userspace client to experiment with the GEOM class. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
VFS locking questions
Hi, I have a couple of questions regarding VFS, since I'm trying to SMPify the fdescfs code in an effort to get some experience with VFS and freebsd locking... What is really LK_INTERLOCK? When should it be used? When should one acquire it (with VI_LOCK i assume), and what are the semantics? Let's say I have a function that should return a locked vnode. I lock the hash-table with a regular mutex. Then, when I traverse the list, I check if the entry is what I look for. If it is, I call VI_LOCK() on the vnode, use vget to increment refcount, and then use vn_lock(vp, LK_EXCLUSIVE...) to lock the vnode before the function returns. Is this correct behaviour? The LK_INTERLOCK bothers me a bit, because I'm not 100% sure on how it works. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Formatting time in kernel
On Tue, May 30, 2006 at 07:17:45PM -0500, Rick C. Petty wrote: On Fri, May 26, 2006 at 10:59:18PM +0200, Ulf Lilleengen wrote: I've been looking through the kernel code the past few days, but I have not found what I'm looking for, which is a way to format struct timeval for output in the same matter as the ctime(3) in the standard libc. I keep thinking how this part of the code should not be in kernel because of this, but things will be vastly more complicated if not because of the whole gvinum structure. (This is for use in the gvinum dumpconfig option I'm working on, and I'm not sure if it's really that important showing the creation time, but that's a different discussion). Why can't you just pass the struct timeval up to the userland tool and have gvinum call ctime? Or at the very least pass up a time_t. Maybe go the other route and pass up the whole gv_label (I'm assuming this is the structure you're talking about).. /sbin/gvinum already has to include vinum/geom_vinum_var.h so it's not like it's very kernel-specific (e.g. #ifdef KERNEL). I personally feel that stuff like this belongs in userland not kernel, parsing or converting times, etc. I agree, but currently the output list, printconfig and those are heavily formatted in the kernel, so I was just looking for the easy way out as a temporate solution. However, I intend to pass the whole configuration to userland, not just the label, and let the tools format it there, as I see to this point see no reason for it to be in the kernel. -- Mvh Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Formatting time in kernel
Hi, I've been looking through the kernel code the past few days, but I have not found what I'm looking for, which is a way to format struct timeval for output in the same matter as the ctime(3) in the standard libc. I keep thinking how this part of the code should not be in kernel because of this, but things will be vastly more complicated if not because of the whole gvinum structure. (This is for use in the gvinum dumpconfig option I'm working on, and I'm not sure if it's really that important showing the creation time, but that's a different discussion). I could ofcourse write my own, but I thought I'd just ask here. Thanks in advance -- Mvh Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Contributing to FreeBSD list of things todo
On Tue, Nov 01, 2005 at 02:58:48PM -0800, Brooks Davis wrote: On Tue, Nov 01, 2005 at 04:30:00PM -0600, Ben Siemon wrote: I have a suggestion for things dev people could do to help out with code already done. I noticed the suggestion for compiling with -Wall enabled. Would it serve any purpose to compile the sources with -ansi and or -pedantic as well? I am fairly new to FreeBSD so forgive me if this has alread been addressed. I think the todo list is dated in this area. We now compile many files with large sets of warning flags via the WARNS variable in Makefiles. It is useful to expand the coverage in this area, but it's not all that trivial. It's often fairly easy to make the warning go away without fixing the real problem the warning represented. This means that far too many patches to raise warning levels are useless and waste developer time. There's also the fact that such changes need to be tested on multiple architectures because certain warnings are platform specfic. At the least testing is needed on i386, alpha, and sparc. I've seen an effort to remove these warnings by setting WARNS?=6 . Would this be preferable to work on, on freebsd as well? Ofcourse testing on most architectures. -- Ulf Lilleengen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
