Re: Alan Cox quote? (was: Re: accounting for threads)
Hello, > I am happy that processes in Linux are so marvelous. Linux does not need > a decent POSIX threads implementation because the same functionality can > be achived with processes. Do what you like, you write the kernel code. > I could write my soft using fork special fetaures in Linux. > But I want it to be portable. If threads in linux are so bad, it is bad > luck for me. I will go slow. It its the only portable way todo afordable > shared memory threading without filling your program of shm-. AFAIK, there is a POSIX thread library (libpthread) for Linux, that wraps clone calls in a way portable to other unices. It uses processes (with memory sharing) from kernel point of view, but should look like POSIX threads from application point of view. --- Jan 'Bulb' Hudec <[EMAIL PROTECTED]> - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Hello, I am happy that processes in Linux are so marvelous. Linux does not need a decent POSIX threads implementation because the same functionality can be achived with processes. Do what you like, you write the kernel code. I could write my soft using fork special fetaures in Linux. But I want it to be portable. If threads in linux are so bad, it is bad luck for me. I will go slow. It its the only portable way todo afordable shared memory threading without filling your program of shm-. AFAIK, there is a POSIX thread library (libpthread) for Linux, that wraps clone calls in a way portable to other unices. It uses processes (with memory sharing) from kernel point of view, but should look like POSIX threads from application point of view. --- Jan 'Bulb' Hudec [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
This discussion seems to go nowhere. Thanks for your comments. I know much more on Linux than before. I am happy that processes in Linux are so marvelous. Linux does not need a decent POSIX threads implementation because the same functionality can be achived with processes. Do what you like, you write the kernel code. I could write my soft using fork special fetaures in Linux. But I want it to be portable. If threads in linux are so bad, it is bad luck for me. I will go slow. It its the only portable way todo afordable shared memory threading without filling your program of shm-. -- J.A. Magallon # Let the source be with you... mailto:[EMAIL PROTECTED] Mandrake Linux release 8.1 (Cooker) for i586 Linux werewolf 2.4.5-ac17 #2 SMP Fri Jun 22 01:36:07 CEST 2001 i686 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
This discussion seems to go nowhere. Thanks for your comments. I know much more on Linux than before. I am happy that processes in Linux are so marvelous. Linux does not need a decent POSIX threads implementation because the same functionality can be achived with processes. Do what you like, you write the kernel code. I could write my soft using fork special fetaures in Linux. But I want it to be portable. If threads in linux are so bad, it is bad luck for me. I will go slow. It its the only portable way todo afordable shared memory threading without filling your program of shm-. -- J.A. Magallon # Let the source be with you... mailto:[EMAIL PROTECTED] Mandrake Linux release 8.1 (Cooker) for i586 Linux werewolf 2.4.5-ac17 #2 SMP Fri Jun 22 01:36:07 CEST 2001 i686 - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Sunday 24 June 2001 19:50, Larry McVoy wrote: > On Mon, Jun 25, 2001 at 12:30:02AM +0200, J . A . Magallon wrote: > > They use fork(). > > They port their app to solaris. > > The performance sucks. > > It is not Solaris fault. > > It is linux fast fork() ... > > One for the quotes page, eh? We're terribly sorry, we'll get busy on > adding some delay loops in Linux so it too can be slow. I'm still working that one out myself... Okay, so Linux programmers are supposed to avoid APIs that are either broken, badly implemented, or missing, on some other platform. That's pretty much all of them, isn't it? Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Mon, Jun 25, 2001 at 12:30:02AM +0200, J . A . Magallon wrote: > shell scripting, for example. Or multithreaded web servers. With the above > test (fork() + immediate exec()), you just try to mesaure the speed of fork(). The benchmark that used lmbench that was posted tested fork follwed fork by exit not follwed by exec. Although you could have used vfork in that test, the point of the test was to get some feel for how much overhead the fork and exit together had. A real application could not use vfork, because it would do some actual work before the exit. > Say you have a fork'ing server. On each connection you fork and exec the > real transfer program. Most forking servers I know of don't exec programs to deal with accepted connections. (The only exception I can think of is inetd and its cousins, the performance of which is not important.) >There time for fork matters. It can run very fast > in Linux but suck in other systems. Just because the programmer chose fork() > instead of vfork(). vfork is not portable. There are standards that specify what it does, but the portable semantics are so useless that if you use it you will almost certainly wind up with undefined behavior. However, the original discussion was about threads. How to do fork+exec (or vfork+exec if you insist on using that broken interface) quickly is beside the point. > >> The clean battle should be linux fork-exec vs vfork-exec in > >> Solaris, because for in linux is really a vfork in solaris. > >But the point of threading is it's a fork WITHOUT an exec. > Not always, see above. You appear to be saying that spawning child programs is an example of threading. I don't think that's what most people mean by threading. Galen - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Sunday 24 June 2001 18:30, J . A . Magallon wrote: > Take a programmer comming from other system to linux. If he wants multi- > threading and protable code, he will choose pthreads. And you say to him: > do it with 'clone', it is better. Answer: non protable. Again: do it > with fork(), it is fast in linux. Answer: better for linux, but it is a > real pain in other systems. So we should be as bad as other systems? I'm trying to figure out what you're saying here... > And worst, you are allowing people to program based on a tool that will > give VERY diferent performance when ported to other systems. They use > fork(). They port their app to solaris. The performance sucks. It is not > Solaris fault. It is linux fast fork() that makes people not looking for > the correct standard tool for what they want todo. Hang on, the fact the LInux implementation of something is 10x faster than somebody else's brain damaged implementation is Linux's fault? On Linux, there is no real difference between threads and processes. On some other systems, there is unnecessary overhead for processes. "Threads" are from a programming perspective multiple points of execution that share an awful lot of state. Which basically, processes can be if you tell them to share this state. >From a system perspective threads are a hack to avoid a lot of the overhead processes have poisoning caches and setting up and tearing down page tables and such. Linux processes don't have this level of overhead. So -ON LINUX- there isn't much difference between threads and processes. Because the process overhead threads try to avoid has been optimized away, and thus there is a real world implementation of the two concepts which behaves in an extremely similar manner. Thus the conceptual difference between the two is mostly either artificial or irrelevant, certainly in this context. If there is a context without this difference, then in other contexts it would appear to be an artifact of implementation. On Solaris there is a difference between threads and processes, but the reason is that Solaris' processes are inefficient (and shown by the benchmarks you don't seem to like). > Instead of trying to enhance linux thread support, you try to move > programmes to use linux 'specialities'. No, I'm saying Linux processes right now have 90% of the features of thread support as is, and the remaining differences are mostly semantic. It would be fairly easy to take an API like the OS/2 threading model and put it on Linux. From what I remember of the OS/2 threading model, we're talking about a couple hours worth of work. Adding support for the Posix model is harder, but from what I've seen of the Posix model this is because it's evil and requires a lot of things that don't really have much to do with the concept of threading. I will admit that Linux (or the traditional unix programming model) has some things that don't mesh well with the abstract thread programming model. Signals, for example. But since the traditional abstract threads model doesn't try to handle signals at ALL that I am aware of (OS/2 didn't, Dunno about NT. Posix has some strange duct-tape solution I'm not entirely familiar with... In java you can kill or suspend a thread which is IMPLEMENTED with signals but conceptually discrete...) > >> That is comparing apples and oranges. > > > >Show me a real-world program that makes the distinction you're making. > >Something in actual use. > > shell scripting, for example. You've done multi-threaded shell scripting? I tried to use bash with multiple processes and ran into the fact that $$ always gives the root process's PID, and nobody had ever apparently decided that this was dumb behavior... > Or multithreaded web servers. Apache has a thread pool, you know. Keeps the suckers around... > With the above > test (fork() + immediate exec()), you just try to mesaure the speed of > fork(). Say you have a fork'ing server. On each connection you fork and > exec the real transfer program. Like a CGI script, you mean? Not using mod_perl or mode_php but shelling out? And you're worried about performance? > There time for fork matters. It can run > very fast in Linux but suck in other systems. And this is linux's problem, is it? > Just because the programmer > chose fork() instead of vfork(). So "#define fork vfork" would not be part of the "#ifdef slowaris" block then? I'm still a bit unclear about what the behavior differences of them actualy are. I believe vfork is the hack that goes really funny if you DON'T exec immediately after calling it. Yet the argument here was about THREADS, which don't DO that. So the entire discussion is a red herring. If you're going to fork and then exec, you're not having multiple points of execution in the same program. Linux can do this efficiently on a process level because its fork is efficient (and it has clone() which allows more control
Re: Alan Cox quote? (was: Re: accounting for threads)
> BTW, after all I have read all POSIX threads library should be no more than > a wrapper over fork(), clone and so on. Why are they so bad then ? > I am going to get glibc source to see what is inside pthread_create... If I recall it had to do with problems in signal delivery... -- Gerhard Mack [EMAIL PROTECTED] <>< As a computer I find your faith in technology amusing. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On 20010625 Larry McVoy wrote: > >One for the quotes page, eh? We're terribly sorry, we'll get busy on adding >some delay loops in Linux so it too can be slow. >-- I was afraid someone would tell that... I just want to say that the 'problem' is not that threads are slow in linux, but that others ways are faster than they 'should be if done the standard way'. BTW, after all I have read all POSIX threads library should be no more than a wrapper over fork(), clone and so on. Why are they so bad then ? I am going to get glibc source to see what is inside pthread_create... -- J.A. Magallon # Let the source be with you... mailto:[EMAIL PROTECTED] Mandrake Linux release 8.1 (Cooker) for i586 Linux werewolf 2.4.5-ac17 #2 SMP Fri Jun 22 01:36:07 CEST 2001 i686 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Mon, Jun 25, 2001 at 12:30:02AM +0200, J . A . Magallon wrote: > They use fork(). > They port their app to solaris. > The performance sucks. > It is not Solaris fault. > It is linux fast fork() ... One for the quotes page, eh? We're terribly sorry, we'll get busy on adding some delay loops in Linux so it too can be slow. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Mon, Jun 25, 2001 at 12:30:02AM +0200, J . A . Magallon wrote: > Take a programmer comming from other system to linux. If he wants multi- > threading and protable code, he will choose pthreads. And you say to him: > do it with 'clone', it is better. Answer: non protable. Again: do it > with fork(), it is fast in linux. Answer: better for linux, but it is a > real pain in other systems. > > And worst, you are allowing people to program based on a tool that will give > VERY diferent performance when ported to other systems. They use fork(). > They port their app to solaris. The performance sucks. It is not Solaris > fault. It is linux fast fork() that makes people not looking for the > correct standard tool for what they want todo. This sounds to my like "Linux is making other OSes look bad. Cut it out." -- -Steven In a time of universal deceit, telling the truth is a revolutionary act. -- George Orwell - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On 20010624 Rob Landley wrote: > >This is a bit like like saying that a truck and a train are totally different >beasts. If I'm trying to haul cargo from point A to point B, which is served >by both, all I care about is how long it takes and how much it costs. > >I don't care what it was INTENDED to do. A rock makes a decent hammer. So >does a crescent wrench. The question is how good a tool is it for the uses >we're trying to put it to? > Well, the problem is that when somebody says you it has a truck to take your cargo, you have your idea of a truck, and there is a standard idea of the difference between a train and a truck. Tools have been developed for multi-processing in unix, processes and threads. And have a difference. And when you design your algorithm (and more if you want it to be protable), you think if you want it done with processes or you need threads (posix, or lwps or whatever). And you decide it on the standard idea of what a process and a thread are. Take a programmer comming from other system to linux. If he wants multi- threading and protable code, he will choose pthreads. And you say to him: do it with 'clone', it is better. Answer: non protable. Again: do it with fork(), it is fast in linux. Answer: better for linux, but it is a real pain in other systems. And worst, you are allowing people to program based on a tool that will give VERY diferent performance when ported to other systems. They use fork(). They port their app to solaris. The performance sucks. It is not Solaris fault. It is linux fast fork() that makes people not looking for the correct standard tool for what they want todo. Instead of trying to enhance linux thread support, you try to move programmes to use linux 'specialities'. >> This remembers on other question I read in this thread (I tried to answer >> then but I had broke balsa...). Somebody posted some benchmarks of linux >> fork()+exec() vs Solaris fork()+exec(). > >What programs does this make a difference in? These are tools meant to be >used. What real-world usage causes them to differ? If a reasonably >competent programmer ported a program from one platform to another, is this >the behavior they would expect to see (without a major rewrite of the code)? > >> That is comparing apples and oranges. > >Show me a real-world program that makes the distinction you're making. >Something in actual use. > shell scripting, for example. Or multithreaded web servers. With the above test (fork() + immediate exec()), you just try to mesaure the speed of fork(). Say you have a fork'ing server. On each connection you fork and exec the real transfer program. There time for fork matters. It can run very fast in Linux but suck in other systems. Just because the programmer chose fork() instead of vfork(). >> The clean battle should be linux fork-exec vs vfork-exec in >> Solaris, because for in linux is really a vfork in solaris. > >But the point of threading is it's a fork WITHOUT an exec. > Not always, see above. > >I'm not going to comment on the difference between fork and vfork becaue to >be honest I've forgotten what the difference is. Something to do with Time. -- J.A. Magallon # Let the source be with you... mailto:[EMAIL PROTECTED] Mandrake Linux release 8.1 (Cooker) for i586 Linux werewolf 2.4.5-ac17 #2 SMP Fri Jun 22 01:36:07 CEST 2001 i686 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Sunday 24 June 2001 17:41, J . A . Magallon wrote: > On 20010622 Rob Landley wrote: > >I still consider the difference between threads and processes with shared > >resources (memory, fds, etc) to be largely semantic. > > They should not be the same. Processes are processes, and threads were > designed for situations where processes are too heavy. Other thing is that > in new kernels (for example, Linux) processes are being optimized (ie, vm > fast 'cloning' via copy-on-write) or expanded with new features (Linux' > clone+ CLONE_VM). But they are different beasts. This is a bit like like saying that a truck and a train are totally different beasts. If I'm trying to haul cargo from point A to point B, which is served by both, all I care about is how long it takes and how much it costs. I don't care what it was INTENDED to do. A rock makes a decent hammer. So does a crescent wrench. The question is how good a tool is it for the uses we're trying to put it to? > This remembers on other question I read in this thread (I tried to answer > then but I had broke balsa...). Somebody posted some benchmarks of linux > fork()+exec() vs Solaris fork()+exec(). What programs does this make a difference in? These are tools meant to be used. What real-world usage causes them to differ? If a reasonably competent programmer ported a program from one platform to another, is this the behavior they would expect to see (without a major rewrite of the code)? > That is comparing apples and oranges. Show me a real-world program that makes the distinction you're making. Something in actual use. > The clean battle should be linux fork-exec vs vfork-exec in > Solaris, because for in linux is really a vfork in solaris. But the point of threading is it's a fork WITHOUT an exec. So now you're saying the comparison we're making of "using these tools to do goal X" is invalid because you don't like goal X, and you want to do something else with them. I'm not going to comment on the difference between fork and vfork becaue to be honest I've forgotten what the difference is. Something to do with resource sharing but darned if I can remember it. I'm not sure I've ever used vfork, last thing I remember were people arguing over the man page for it... (Sorry if I seem a bit snappish, I've had this kind of discussion too many times with ivory tower types at UT who object to the topic of conversation because it's not what they've focused on until now. Which logical fallacy was this, "begging the question"? Straw man? I forget. I'm tired...) Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On 20010622 Rob Landley wrote: > >I still consider the difference between threads and processes with shared >resources (memory, fds, etc) to be largely semantic. > They should not be the same. Processes are processes, and threads were designed for situations where processes are too heavy. Other thing is that in new kernels (for example, Linux) processes are being optimized (ie, vm fast 'cloning' via copy-on-write) or expanded with new features (Linux' clone+ CLONE_VM). But they are different beasts. This remembers on other question I read in this thread (I tried to answer then but I had broke balsa...). Somebody posted some benchmarks of linux fork()+exec() vs Solaris fork()+exec(). That is comparing apples and oranges. The clean battle should be linux fork-exec vs vfork-exec in Solaris, because for in linux is really a vfork in solaris. -- J.A. Magallon # Let the source be with you... mailto:[EMAIL PROTECTED] Mandrake Linux release 8.1 (Cooker) for i586 Linux werewolf 2.4.5-ac17 #2 SMP Fri Jun 22 01:36:07 CEST 2001 i686 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On 20010622 Rob Landley wrote: I still consider the difference between threads and processes with shared resources (memory, fds, etc) to be largely semantic. They should not be the same. Processes are processes, and threads were designed for situations where processes are too heavy. Other thing is that in new kernels (for example, Linux) processes are being optimized (ie, vm fast 'cloning' via copy-on-write) or expanded with new features (Linux' clone+ CLONE_VM). But they are different beasts. This remembers on other question I read in this thread (I tried to answer then but I had broke balsa...). Somebody posted some benchmarks of linux fork()+exec() vs Solaris fork()+exec(). That is comparing apples and oranges. The clean battle should be linux fork-exec vs vfork-exec in Solaris, because for in linux is really a vfork in solaris. -- J.A. Magallon # Let the source be with you... mailto:[EMAIL PROTECTED] Mandrake Linux release 8.1 (Cooker) for i586 Linux werewolf 2.4.5-ac17 #2 SMP Fri Jun 22 01:36:07 CEST 2001 i686 - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Sunday 24 June 2001 17:41, J . A . Magallon wrote: On 20010622 Rob Landley wrote: I still consider the difference between threads and processes with shared resources (memory, fds, etc) to be largely semantic. They should not be the same. Processes are processes, and threads were designed for situations where processes are too heavy. Other thing is that in new kernels (for example, Linux) processes are being optimized (ie, vm fast 'cloning' via copy-on-write) or expanded with new features (Linux' clone+ CLONE_VM). But they are different beasts. This is a bit like like saying that a truck and a train are totally different beasts. If I'm trying to haul cargo from point A to point B, which is served by both, all I care about is how long it takes and how much it costs. I don't care what it was INTENDED to do. A rock makes a decent hammer. So does a crescent wrench. The question is how good a tool is it for the uses we're trying to put it to? This remembers on other question I read in this thread (I tried to answer then but I had broke balsa...). Somebody posted some benchmarks of linux fork()+exec() vs Solaris fork()+exec(). What programs does this make a difference in? These are tools meant to be used. What real-world usage causes them to differ? If a reasonably competent programmer ported a program from one platform to another, is this the behavior they would expect to see (without a major rewrite of the code)? That is comparing apples and oranges. Show me a real-world program that makes the distinction you're making. Something in actual use. The clean battle should be linux fork-exec vs vfork-exec in Solaris, because for in linux is really a vfork in solaris. But the point of threading is it's a fork WITHOUT an exec. So now you're saying the comparison we're making of using these tools to do goal X is invalid because you don't like goal X, and you want to do something else with them. I'm not going to comment on the difference between fork and vfork becaue to be honest I've forgotten what the difference is. Something to do with resource sharing but darned if I can remember it. I'm not sure I've ever used vfork, last thing I remember were people arguing over the man page for it... (Sorry if I seem a bit snappish, I've had this kind of discussion too many times with ivory tower types at UT who object to the topic of conversation because it's not what they've focused on until now. Which logical fallacy was this, begging the question? Straw man? I forget. I'm tired...) Rob - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On 20010624 Rob Landley wrote: This is a bit like like saying that a truck and a train are totally different beasts. If I'm trying to haul cargo from point A to point B, which is served by both, all I care about is how long it takes and how much it costs. I don't care what it was INTENDED to do. A rock makes a decent hammer. So does a crescent wrench. The question is how good a tool is it for the uses we're trying to put it to? Well, the problem is that when somebody says you it has a truck to take your cargo, you have your idea of a truck, and there is a standard idea of the difference between a train and a truck. Tools have been developed for multi-processing in unix, processes and threads. And have a difference. And when you design your algorithm (and more if you want it to be protable), you think if you want it done with processes or you need threads (posix, or lwps or whatever). And you decide it on the standard idea of what a process and a thread are. Take a programmer comming from other system to linux. If he wants multi- threading and protable code, he will choose pthreads. And you say to him: do it with 'clone', it is better. Answer: non protable. Again: do it with fork(), it is fast in linux. Answer: better for linux, but it is a real pain in other systems. And worst, you are allowing people to program based on a tool that will give VERY diferent performance when ported to other systems. They use fork(). They port their app to solaris. The performance sucks. It is not Solaris fault. It is linux fast fork() that makes people not looking for the correct standard tool for what they want todo. Instead of trying to enhance linux thread support, you try to move programmes to use linux 'specialities'. This remembers on other question I read in this thread (I tried to answer then but I had broke balsa...). Somebody posted some benchmarks of linux fork()+exec() vs Solaris fork()+exec(). What programs does this make a difference in? These are tools meant to be used. What real-world usage causes them to differ? If a reasonably competent programmer ported a program from one platform to another, is this the behavior they would expect to see (without a major rewrite of the code)? That is comparing apples and oranges. Show me a real-world program that makes the distinction you're making. Something in actual use. shell scripting, for example. Or multithreaded web servers. With the above test (fork() + immediate exec()), you just try to mesaure the speed of fork(). Say you have a fork'ing server. On each connection you fork and exec the real transfer program. There time for fork matters. It can run very fast in Linux but suck in other systems. Just because the programmer chose fork() instead of vfork(). The clean battle should be linux fork-exec vs vfork-exec in Solaris, because for in linux is really a vfork in solaris. But the point of threading is it's a fork WITHOUT an exec. Not always, see above. I'm not going to comment on the difference between fork and vfork becaue to be honest I've forgotten what the difference is. Something to do with Time. -- J.A. Magallon # Let the source be with you... mailto:[EMAIL PROTECTED] Mandrake Linux release 8.1 (Cooker) for i586 Linux werewolf 2.4.5-ac17 #2 SMP Fri Jun 22 01:36:07 CEST 2001 i686 - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Mon, Jun 25, 2001 at 12:30:02AM +0200, J . A . Magallon wrote: Take a programmer comming from other system to linux. If he wants multi- threading and protable code, he will choose pthreads. And you say to him: do it with 'clone', it is better. Answer: non protable. Again: do it with fork(), it is fast in linux. Answer: better for linux, but it is a real pain in other systems. And worst, you are allowing people to program based on a tool that will give VERY diferent performance when ported to other systems. They use fork(). They port their app to solaris. The performance sucks. It is not Solaris fault. It is linux fast fork() that makes people not looking for the correct standard tool for what they want todo. This sounds to my like Linux is making other OSes look bad. Cut it out. -- -Steven In a time of universal deceit, telling the truth is a revolutionary act. -- George Orwell - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Mon, Jun 25, 2001 at 12:30:02AM +0200, J . A . Magallon wrote: They use fork(). They port their app to solaris. The performance sucks. It is not Solaris fault. It is linux fast fork() ... One for the quotes page, eh? We're terribly sorry, we'll get busy on adding some delay loops in Linux so it too can be slow. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On 20010625 Larry McVoy wrote: One for the quotes page, eh? We're terribly sorry, we'll get busy on adding some delay loops in Linux so it too can be slow. -- I was afraid someone would tell that... I just want to say that the 'problem' is not that threads are slow in linux, but that others ways are faster than they 'should be if done the standard way'. BTW, after all I have read all POSIX threads library should be no more than a wrapper over fork(), clone and so on. Why are they so bad then ? I am going to get glibc source to see what is inside pthread_create... -- J.A. Magallon # Let the source be with you... mailto:[EMAIL PROTECTED] Mandrake Linux release 8.1 (Cooker) for i586 Linux werewolf 2.4.5-ac17 #2 SMP Fri Jun 22 01:36:07 CEST 2001 i686 - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
BTW, after all I have read all POSIX threads library should be no more than a wrapper over fork(), clone and so on. Why are they so bad then ? I am going to get glibc source to see what is inside pthread_create... If I recall it had to do with problems in signal delivery... -- Gerhard Mack [EMAIL PROTECTED] As a computer I find your faith in technology amusing. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Sunday 24 June 2001 18:30, J . A . Magallon wrote: Take a programmer comming from other system to linux. If he wants multi- threading and protable code, he will choose pthreads. And you say to him: do it with 'clone', it is better. Answer: non protable. Again: do it with fork(), it is fast in linux. Answer: better for linux, but it is a real pain in other systems. So we should be as bad as other systems? I'm trying to figure out what you're saying here... And worst, you are allowing people to program based on a tool that will give VERY diferent performance when ported to other systems. They use fork(). They port their app to solaris. The performance sucks. It is not Solaris fault. It is linux fast fork() that makes people not looking for the correct standard tool for what they want todo. Hang on, the fact the LInux implementation of something is 10x faster than somebody else's brain damaged implementation is Linux's fault? On Linux, there is no real difference between threads and processes. On some other systems, there is unnecessary overhead for processes. Threads are from a programming perspective multiple points of execution that share an awful lot of state. Which basically, processes can be if you tell them to share this state. From a system perspective threads are a hack to avoid a lot of the overhead processes have poisoning caches and setting up and tearing down page tables and such. Linux processes don't have this level of overhead. So -ON LINUX- there isn't much difference between threads and processes. Because the process overhead threads try to avoid has been optimized away, and thus there is a real world implementation of the two concepts which behaves in an extremely similar manner. Thus the conceptual difference between the two is mostly either artificial or irrelevant, certainly in this context. If there is a context without this difference, then in other contexts it would appear to be an artifact of implementation. On Solaris there is a difference between threads and processes, but the reason is that Solaris' processes are inefficient (and shown by the benchmarks you don't seem to like). Instead of trying to enhance linux thread support, you try to move programmes to use linux 'specialities'. No, I'm saying Linux processes right now have 90% of the features of thread support as is, and the remaining differences are mostly semantic. It would be fairly easy to take an API like the OS/2 threading model and put it on Linux. From what I remember of the OS/2 threading model, we're talking about a couple hours worth of work. Adding support for the Posix model is harder, but from what I've seen of the Posix model this is because it's evil and requires a lot of things that don't really have much to do with the concept of threading. I will admit that Linux (or the traditional unix programming model) has some things that don't mesh well with the abstract thread programming model. Signals, for example. But since the traditional abstract threads model doesn't try to handle signals at ALL that I am aware of (OS/2 didn't, Dunno about NT. Posix has some strange duct-tape solution I'm not entirely familiar with... In java you can kill or suspend a thread which is IMPLEMENTED with signals but conceptually discrete...) That is comparing apples and oranges. Show me a real-world program that makes the distinction you're making. Something in actual use. shell scripting, for example. You've done multi-threaded shell scripting? I tried to use bash with multiple processes and ran into the fact that $$ always gives the root process's PID, and nobody had ever apparently decided that this was dumb behavior... Or multithreaded web servers. Apache has a thread pool, you know. Keeps the suckers around... With the above test (fork() + immediate exec()), you just try to mesaure the speed of fork(). Say you have a fork'ing server. On each connection you fork and exec the real transfer program. Like a CGI script, you mean? Not using mod_perl or mode_php but shelling out? And you're worried about performance? There time for fork matters. It can run very fast in Linux but suck in other systems. And this is linux's problem, is it? Just because the programmer chose fork() instead of vfork(). So #define fork vfork would not be part of the #ifdef slowaris block then? I'm still a bit unclear about what the behavior differences of them actualy are. I believe vfork is the hack that goes really funny if you DON'T exec immediately after calling it. Yet the argument here was about THREADS, which don't DO that. So the entire discussion is a red herring. If you're going to fork and then exec, you're not having multiple points of execution in the same program. Linux can do this efficiently on a process level because its fork is efficient (and it has clone() which allows more control over what gets shared between processes.)
Re: Alan Cox quote? (was: Re: accounting for threads)
On Mon, Jun 25, 2001 at 12:30:02AM +0200, J . A . Magallon wrote: shell scripting, for example. Or multithreaded web servers. With the above test (fork() + immediate exec()), you just try to mesaure the speed of fork(). The benchmark that used lmbench that was posted tested fork follwed fork by exit not follwed by exec. Although you could have used vfork in that test, the point of the test was to get some feel for how much overhead the fork and exit together had. A real application could not use vfork, because it would do some actual work before the exit. Say you have a fork'ing server. On each connection you fork and exec the real transfer program. Most forking servers I know of don't exec programs to deal with accepted connections. (The only exception I can think of is inetd and its cousins, the performance of which is not important.) There time for fork matters. It can run very fast in Linux but suck in other systems. Just because the programmer chose fork() instead of vfork(). vfork is not portable. There are standards that specify what it does, but the portable semantics are so useless that if you use it you will almost certainly wind up with undefined behavior. However, the original discussion was about threads. How to do fork+exec (or vfork+exec if you insist on using that broken interface) quickly is beside the point. The clean battle should be linux fork-exec vs vfork-exec in Solaris, because for in linux is really a vfork in solaris. But the point of threading is it's a fork WITHOUT an exec. Not always, see above. You appear to be saying that spawning child programs is an example of threading. I don't think that's what most people mean by threading. Galen - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Sunday 24 June 2001 19:50, Larry McVoy wrote: On Mon, Jun 25, 2001 at 12:30:02AM +0200, J . A . Magallon wrote: They use fork(). They port their app to solaris. The performance sucks. It is not Solaris fault. It is linux fast fork() ... One for the quotes page, eh? We're terribly sorry, we'll get busy on adding some delay loops in Linux so it too can be slow. I'm still working that one out myself... Okay, so Linux programmers are supposed to avoid APIs that are either broken, badly implemented, or missing, on some other platform. That's pretty much all of them, isn't it? Rob - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Friday 22 June 2001 10:46, Mikulas Patocka wrote: > I did some threaded programming on OS/2 and it was real pain. The main > design flaw in OS/2 API is that thread can be blocked only on one > condition. There is no way thread can wait for more events. For example Sure. But you know what a race condition is, and how to spot one (in potential during coding, or during debugging.) You know how to use semaphores and when and why, and when you DON'T need them. You know about the potential for deadlocks. And most of all, you know just because you got it to run once doesn't mean it's RIGHT... > When OS/2 designers realised this API braindamage, they somewhere randomly > added funtions to unblock threads waiting for variuos events - for example > VioModeUndo or VioSavRedrawUndo - quite insane. OS/2 had a whole raft of problems. The fact half the system calls weren't available if you didn't boot the GUI was my personal favorite annoyance. It was a system created _for_ users instead of _by_ users. Think of the great successes in the computing world: C, Unix, the internet, the web. All of them were developed by people who were just trying to use them, as the tools they used which they modified and extended in response to their needs. This is why C is a better language than pascal, why the internet beat compuserve, and why Unix was better than OS/2. Third parties writing code "for" somebody else (to sell, as a teaching tool, etc) either leave important stuff out or add in stuff people don't want (featuritis). It's the nature of the beast: design may be clever in spurts but evolution never sleeps. (Anybody who doesn't believe that has never studied antibiotic resistant bacteria, or had to deal with cockroaches.) > Programming with select, poll and kqueue on Unix is really much better > than with threads on OS/2. I still consider the difference between threads and processes with shared resources (memory, fds, etc) to be largely semantic. > Mikulas Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Friday 22 June 2001 10:46, Mikulas Patocka wrote: I did some threaded programming on OS/2 and it was real pain. The main design flaw in OS/2 API is that thread can be blocked only on one condition. There is no way thread can wait for more events. For example Sure. But you know what a race condition is, and how to spot one (in potential during coding, or during debugging.) You know how to use semaphores and when and why, and when you DON'T need them. You know about the potential for deadlocks. And most of all, you know just because you got it to run once doesn't mean it's RIGHT... When OS/2 designers realised this API braindamage, they somewhere randomly added funtions to unblock threads waiting for variuos events - for example VioModeUndo or VioSavRedrawUndo - quite insane. OS/2 had a whole raft of problems. The fact half the system calls weren't available if you didn't boot the GUI was my personal favorite annoyance. It was a system created _for_ users instead of _by_ users. Think of the great successes in the computing world: C, Unix, the internet, the web. All of them were developed by people who were just trying to use them, as the tools they used which they modified and extended in response to their needs. This is why C is a better language than pascal, why the internet beat compuserve, and why Unix was better than OS/2. Third parties writing code for somebody else (to sell, as a teaching tool, etc) either leave important stuff out or add in stuff people don't want (featuritis). It's the nature of the beast: design may be clever in spurts but evolution never sleeps. (Anybody who doesn't believe that has never studied antibiotic resistant bacteria, or had to deal with cockroaches.) Programming with select, poll and kqueue on Unix is really much better than with threads on OS/2. I still consider the difference between threads and processes with shared resources (memory, fds, etc) to be largely semantic. Mikulas Rob - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Alexander Viro writes: > BTW, proc_net_create() is also not a good idea if you block the > interrupts. Ditto for netlink_kernel_create(), AFAICS (due to > netlink_kernel_creat() -> sock_alloc() -> get_empty_inode() -> > kmem_cache_alloc() with SLAB_KERNEL). > > That, BTW, is a nice illustration - it's easy to get a preemption > point without noticing, so holding spinlocks, let alone disabling > interrupts over the large area is going to hurt like hell. Here's an idea: add a CONFIGable debug mode for spinlock/cli interaction with GFP_KERNEL and other (known) blocking operations. Keep a per-CPU flag or bitmask that's manipulated by lock/cli operations and checked by memory allocators and other key blocking operations. Generate an Oops upon violation. Make the CONFIG option initially set to 'y' for a patch level or two. Regards, Richard Permanent: [EMAIL PROTECTED] Current: [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
> > A lot of OS/2 software is written with this feature in mind. I know of one > > programmer who absolutely hates Linux because it's just too difficult > > porting software to it, and the lack of decent thread support is part of > > the problem. > > Yup. OS/2 is the largest nest of trained, experienced multi-threaded > programmers. (And it takes a LOT of training experience to do threads > right.) That's why I've been trying to recruit ex-OS/2 guys over to Linux > for years now. (Most followed Java to Linux after Netscape opened up > Mozilla, but there used to be several notable holdouts...) I did some threaded programming on OS/2 and it was real pain. The main design flaw in OS/2 API is that thread can be blocked only on one condition. There is no way thread can wait for more events. For example I have a thread that processes GUI messages. It is blocked in WinGetMsg - waiting for messages from pmshell. Now another part of application needs to wake up the thread to do some things - but there is no way to wake it up because the thread can't wait for anything else than messages from pmshell. I actually had to create dummy invisible window. I send messages to that GUI thread via event queue of that window - really dirty but it can't be done otherwise. When OS/2 designers realised this API braindamage, they somewhere randomly added funtions to unblock threads waiting for variuos events - for example VioModeUndo or VioSavRedrawUndo - quite insane. Programming with select, poll and kqueue on Unix is really much better than with threads on OS/2. Mikulas - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Alexander Viro <[EMAIL PROTECTED]> writes: > On Thu, 21 Jun 2001, Rusty Russell wrote: > > > Disagree. A significant percentage of the netfilter bugs have been > > SMP only (the whole thing is non-reentrant on UP). > > I really doubt it. > Well, if you use GFP_ATOMIC for everything... grep... > Erm... AFAICS, you call create_chain() with interrupts disabled > (under write_lock_irq_save()). Unless I'm _very_ mistaken, > kmalloc(..., GFP_KERNEL) is a Bad Thing(tm) in that situation. > And create_chain() leads to it. That's the old ipchains code; if it's buggy it's likely buggy in 2.2 too. In fact 2.2 ip_fw.c has exactly the same problem (and also a panic on OOM) iptables is supposed to be somewhat better (that's the new code), with cleaner locking. -Andi, wondering why the Stanford Checker didn't catch that. P.S.: Could you please always change the subject in future when going from slashdot niveau material to real kernel bugs on l-k. thanks. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Alexander Viro [EMAIL PROTECTED] writes: On Thu, 21 Jun 2001, Rusty Russell wrote: Disagree. A significant percentage of the netfilter bugs have been SMP only (the whole thing is non-reentrant on UP). I really doubt it. looking through the thing raised brows Well, if you use GFP_ATOMIC for everything... grep... Erm... AFAICS, you call create_chain() with interrupts disabled (under write_lock_irq_save()). Unless I'm _very_ mistaken, kmalloc(..., GFP_KERNEL) is a Bad Thing(tm) in that situation. And create_chain() leads to it. That's the old ipchains code; if it's buggy it's likely buggy in 2.2 too. In fact 2.2 ip_fw.c has exactly the same problem (and also a panic on OOM) iptables is supposed to be somewhat better (that's the new code), with cleaner locking. -Andi, wondering why the Stanford Checker didn't catch that. P.S.: Could you please always change the subject in future when going from slashdot niveau material to real kernel bugs on l-k. thanks. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
A lot of OS/2 software is written with this feature in mind. I know of one programmer who absolutely hates Linux because it's just too difficult porting software to it, and the lack of decent thread support is part of the problem. Yup. OS/2 is the largest nest of trained, experienced multi-threaded programmers. (And it takes a LOT of training experience to do threads right.) That's why I've been trying to recruit ex-OS/2 guys over to Linux for years now. (Most followed Java to Linux after Netscape opened up Mozilla, but there used to be several notable holdouts...) I did some threaded programming on OS/2 and it was real pain. The main design flaw in OS/2 API is that thread can be blocked only on one condition. There is no way thread can wait for more events. For example I have a thread that processes GUI messages. It is blocked in WinGetMsg - waiting for messages from pmshell. Now another part of application needs to wake up the thread to do some things - but there is no way to wake it up because the thread can't wait for anything else than messages from pmshell. I actually had to create dummy invisible window. I send messages to that GUI thread via event queue of that window - really dirty but it can't be done otherwise. When OS/2 designers realised this API braindamage, they somewhere randomly added funtions to unblock threads waiting for variuos events - for example VioModeUndo or VioSavRedrawUndo - quite insane. Programming with select, poll and kqueue on Unix is really much better than with threads on OS/2. Mikulas - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Alexander Viro writes: BTW, proc_net_create() is also not a good idea if you block the interrupts. Ditto for netlink_kernel_create(), AFAICS (due to netlink_kernel_creat() - sock_alloc() - get_empty_inode() - kmem_cache_alloc() with SLAB_KERNEL). That, BTW, is a nice illustration - it's easy to get a preemption point without noticing, so holding spinlocks, let alone disabling interrupts over the large area is going to hurt like hell. Here's an idea: add a CONFIGable debug mode for spinlock/cli interaction with GFP_KERNEL and other (known) blocking operations. Keep a per-CPU flag or bitmask that's manipulated by lock/cli operations and checked by memory allocators and other key blocking operations. Generate an Oops upon violation. Make the CONFIG option initially set to 'y' for a patch level or two. Regards, Richard Permanent: [EMAIL PROTECTED] Current: [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Rok papez <[EMAIL PROTECTED]> wrote: >On Tuesday 19 June 2001 18:09, Larry McVoy wrote: >> "If you think you need threads then your processes are too fat" >> ``Think of it this way: threads are like salt, not like pasta You >> like salt, I like salt, we all like salt. But we eat more pasta.'' > Here are more from the same basket you obviously got the first > quote from: [SNIP] > Protected memory is a constant 10% CPU hog needed only by undisciplined > programmers who can't keep their memory writes in their own process space. Now that's the primary reason to avoid threads, IMO. People spent years trying to get protected memory; the only real difference between threads and processes is that you throw protected memory out the window. Yes, the pthread API sucks; yes, there are times when threads are appropriate. If only: 1) Microsoft, Sun, and others didn't have such abysmal context switch numbers that people view threads vs. processes as a performance argument; 2) MS didn't conflate fork() and exec() into one CreateProcess() call; and 3) Java and others exposed rational event-driven APIs that didn't require multiple contexts of execution in weird places (1.4 is finally fixing this one anyway) then people might be able to really see that: 1) You usually don't need that many contexts of execution; and 2) Processes should be considered the primary COEs and threads only used when you really need to share almost all of your memory. That's aside from all the arguments against multithreading just based on elegance and code correctness POVs. Even if writing multithreaded code is marginally easier than writing poll()-based code, actually debugging it is a royal PITA. Coroutines (which aren't Alan's, Knuth had them long before Alan and even he was just rehashing old news) and state machines really are better in many cases. About the only criticism I have of Alan's statement that "threads are for people who can't program state machines" is the implication that threads are an easier API to get right. They aren't. They seem that way, but by tossing protected memory and introducing multiple COEs you get into a whole world of non-obvious problems that are very difficult to debug and often nearly impossible to reproduce in non-production environments. The salt quote I like; it allows for the sparing use of threads, but warns against their over(ab)use. Ah, well. Sumner __ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail http://personal.mail.yahoo.com/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On 20 Jun 2001 10:14:48 +0100, Alan Cox wrote: > It does. ... not > They are always readable. That's not very useful. Not in the sense of supporting aync, non-blocking i/o to disk files without using threads. -- Michael Rothwell [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Thu, 21 Jun 2001, Alexander Viro wrote: > > > On Thu, 21 Jun 2001, Rusty Russell wrote: > > > Disagree. A significant percentage of the netfilter bugs have been > > SMP only (the whole thing is non-reentrant on UP). > > I really doubt it. > Well, if you use GFP_ATOMIC for everything... grep... > Erm... AFAICS, you call create_chain() with interrupts disabled > (under write_lock_irq_save()). Unless I'm _very_ mistaken, > kmalloc(..., GFP_KERNEL) is a Bad Thing(tm) in that situation. > And create_chain() leads to it. BTW, proc_net_create() is also not a good idea if you block the interrupts. Ditto for netlink_kernel_create(), AFAICS (due to netlink_kernel_creat() -> sock_alloc() -> get_empty_inode() -> kmem_cache_alloc() with SLAB_KERNEL). That, BTW, is a nice illustration - it's easy to get a preemption point without noticing, so holding spinlocks, let alone disabling interrupts over the large area is going to hurt like hell. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Thu, 21 Jun 2001, Rusty Russell wrote: > Disagree. A significant percentage of the netfilter bugs have been > SMP only (the whole thing is non-reentrant on UP). I really doubt it. Well, if you use GFP_ATOMIC for everything... grep... Erm... AFAICS, you call create_chain() with interrupts disabled (under write_lock_irq_save()). Unless I'm _very_ mistaken, kmalloc(..., GFP_KERNEL) is a Bad Thing(tm) in that situation. And create_chain() leads to it. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
In message <[EMAIL PROTECTED]> you wri te: > In practice it's a BS. There is a lot of ways minor modifications of code > could add a preemption point, so if you rely on the lack of such - expect > major PITA. > > Yes, in theory SMP adds some extra fun. Practically, almost every "SMP" > race found so far did not require SMP. Disagree. A significant percentage of the netfilter bugs have been SMP only (the whole thing is non-reentrant on UP). Depends what part of the kernel you play in, Rusty. -- Premature optmztion is rt of all evl. --DK - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Idea: Patches-from-linus mailing list? (Was Re: Alan Cox quote? (was: Re: accounting for threads))
On Wednesday 20 June 2001 21:57, D. Stimits wrote: > MySQL is just a sample. I mention it because it is quite easy to link a > web server to. Imagine patch running on a large file that is a > conglomeration of 50 small patches; it could easily summarize this, and > storing it through MySQL adds a lot of increased web flexibility (such > as searching and sorting). It is, however, just one example of a way to > make "patch" become autodocumenting. Not so much patch as a wrapper around patch. That's a good idea. A small perl script would do it... Right now, Linus makes a big file by appending mail messages to it. His mailer is, in theory, putting mail headers at the start of each of these messages (from, to, subject, and all that). At the end of the day, he feeds that big file to patch and it applies all the patches he's read through and decided he likes. It should be fairly easy to make a wrapper around patch that splits out a single mail message, feeds it to patch, and on some measurement of "success" forwards it to an otherwise read-only mailing list. (Which can then have a database based archiver subscribed to that list, if necessary.) If Linus used such a beast, we could get the actual mail messages Linus is applying patches from, as they're applied. Including any human readable documentation in them that patch itself would discard. No more asking "did patch such and such get applied, when, who was it from"... And we get the extra patch granularity Linus himself is so keen on. Instead of waiting for our weekly pre2-pre3 100k patch, we could follow the individual ones as logically grouped changes, with subject lines saying what the patch is about and everything. And the people hankering to make a CVS tree out of LInux kernel development would then have a much better checkin granularity to work with. :) The main thing, though, is that done right, it's no extra burden on Linus. (Which is kind of important if we ever hope to get him to use it. :) Sound like an idea to anybody else? Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Thursday 21 June 2001 10:02, Jesse Pollard wrote: > Rob Landley <[EMAIL PROTECTED]>: > > On Wednesday 20 June 2001 17:20, Albert D. Cahalan wrote: > > > Rob Landley writes: > > > > My only real gripe with Linux's threads right now [...] is > > > > that ps and top and such aren't thread aware and don't group them > > > > right. > > > > > > > > I'm told they added some kind of "threadgroup" field to processes > > > > that allows top and ps and such to get the display right. I haven't > > > > noticed any upgrades, and haven't had time to go hunting myself. > > > > > > There was a "threadgroup" added just before the 2.4 release. > > > Linus said he'd remove it if he didn't get comments on how > > > useful it was, examples of usage, etc. So I figured I'd look at > > > the code that weekend, but the patch was removed before then! > > > > Can we give him feedback now, asking him to put it back? > > > > > Submit patches to me, under the LGPL please. The FSF isn't likely > > > to care. What, did you think this was the GNU system or something? > > > > I've stopped even TRYING to patch bash. try a for loop calling "echo > > $$&", eery single process bash forks off has the PARENT'S value for $$, > > which is REALLY annoying if you spend an afternoon writing code not > > knowing that and then wonder why the various process's temp file sare > > stomping each other... > > Actually - you have an error there. $$ gets evaluated during the parse, not > during execution of the subprocess. Well, that would explain it then. (So $$ isn't actually an environment variable, it just looks like one? I had my threads calling a seperate function...) > To get what you are describing it is > necessary to "sh -c 'echo $$'" to force the delay of evaluation. Except that this spawns a new child and gives me the PID of the child, which lives for maybe a tenth of a second... Maybe I could do something with eval...? Either way, I switched the project in question to python. > That depends on what you are trying to do. A couple people emailed me and pointed out I had to set IFS=$'\n', (often after first saying "there's no way to do that" and then correcting themselves towards the end of the message...) My first attempt to do it was to pipe the data into a sub-function do read a line at a time and store the results in an array. (Seemed pretty straightforward.) Except there's no way to get that array back out of the function, and that IS documented at the end of the bash man page. As I said: Python. > Are you trying to echo the > entire "ls -l"? or are you trying to convert an "ls -l" into a single > column based on a field extracted from "ls -l". I was trying to convert the lines into an array I could then iterate through. > If the fields don't matter, but you want each line processed in the > loop do: > > ls -l | while read i > do >echo line:$i > done Hmmm... If that avoids the need to export the array from one shell instance to another, maybe it'd work. (I'm not really doing echo, I was doing var[$i]="line", with some other misc stuff.) But I got it to work via IFS, and that's good enough for right now. > If you want such elaborate handling of strings, I suggest using perl. I came to the same conclusion, but would like for more than one person to be able to work on the same piece of code, so it's python. (Or PHP if Carl's the one who gets around to porting it first. :) But in the meantime, the shell script is now working. Thanks everybody...) Back to talking about the kernel.:) Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Rob Landley <[EMAIL PROTECTED]>:
>
> On Wednesday 20 June 2001 17:20, Albert D. Cahalan wrote:
> > Rob Landley writes:
> > > My only real gripe with Linux's threads right now [...] is
> > > that ps and top and such aren't thread aware and don't group them
> > > right.
> > >
> > > I'm told they added some kind of "threadgroup" field to processes
> > > that allows top and ps and such to get the display right. I haven't
> > > noticed any upgrades, and haven't had time to go hunting myself.
> >
> > There was a "threadgroup" added just before the 2.4 release.
> > Linus said he'd remove it if he didn't get comments on how
> > useful it was, examples of usage, etc. So I figured I'd look at
> > the code that weekend, but the patch was removed before then!
>
> Can we give him feedback now, asking him to put it back?
>
> > Submit patches to me, under the LGPL please. The FSF isn't likely
> > to care. What, did you think this was the GNU system or something?
>
> I've stopped even TRYING to patch bash. try a for loop calling "echo $$&",
> eery single process bash forks off has the PARENT'S value for $$, which is
> REALLY annoying if you spend an afternoon writing code not knowing that and
> then wonder why the various process's temp file sare stomping each other...
Actually - you have an error there. $$ gets evaluated during the parse, not
during execution of the subprocess. To get what you are describing it is
necessary to "sh -c 'echo $$'" to force the delay of evaluation. The only
"bug" interpretation is in the evaluation of the quotes. IF echo '$$' &
delayed the interpretation of "$$", then when the subprocess shell
"echo $$" reparsed the line the $$ would be substituted as you wanted.
This delay can only be done via the "sh -c ..." method. (its the same with
bourne/korn shell).
> Oh, and anybody who can explain this is welcome to try:
>
> lines=`ls -l | awk '{print "\""$0"\""}'`
> for i in $lines
> do
> echo line:$i
> done
That depends on what you are trying to do. Are you trying to echo the
entire "ls -l"? or are you trying to convert an "ls -l" into a single
column based on a field extracted from "ls -l".
If the latter, then the script should be:
ls -l | awk '{print $}' | while read i
do
echo line: $i
done
If the fields don't matter, but you want each line processed in the
loop do:
ls -l | while read i
do
echo line:$i
done
Bash doesn't need patching for this.
Again, the evaluation of the quotes is biting you. When the $lines
parameter is evaluated, the quotes are present.
bash is doing a double evaluation for "for" loop. It expects
a list of single tokens, rather than a list of quoted strings. This is
the same as in the bourne/korn shell.
If you want such elaborate handling of strings, I suggest using perl.
-
Jesse I Pollard, II
Email: [EMAIL PROTECTED]
Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Rob Landley [EMAIL PROTECTED]:
On Wednesday 20 June 2001 17:20, Albert D. Cahalan wrote:
Rob Landley writes:
My only real gripe with Linux's threads right now [...] is
that ps and top and such aren't thread aware and don't group them
right.
I'm told they added some kind of threadgroup field to processes
that allows top and ps and such to get the display right. I haven't
noticed any upgrades, and haven't had time to go hunting myself.
There was a threadgroup added just before the 2.4 release.
Linus said he'd remove it if he didn't get comments on how
useful it was, examples of usage, etc. So I figured I'd look at
the code that weekend, but the patch was removed before then!
Can we give him feedback now, asking him to put it back?
Submit patches to me, under the LGPL please. The FSF isn't likely
to care. What, did you think this was the GNU system or something?
I've stopped even TRYING to patch bash. try a for loop calling echo $$,
eery single process bash forks off has the PARENT'S value for $$, which is
REALLY annoying if you spend an afternoon writing code not knowing that and
then wonder why the various process's temp file sare stomping each other...
Actually - you have an error there. $$ gets evaluated during the parse, not
during execution of the subprocess. To get what you are describing it is
necessary to sh -c 'echo $$' to force the delay of evaluation. The only
bug interpretation is in the evaluation of the quotes. IF echo '$$'
delayed the interpretation of $$, then when the subprocess shell
echo $$ reparsed the line the $$ would be substituted as you wanted.
This delay can only be done via the sh -c ... method. (its the same with
bourne/korn shell).
Oh, and anybody who can explain this is welcome to try:
lines=`ls -l | awk '{print \$0\}'`
for i in $lines
do
echo line:$i
done
That depends on what you are trying to do. Are you trying to echo the
entire ls -l? or are you trying to convert an ls -l into a single
column based on a field extracted from ls -l.
If the latter, then the script should be:
ls -l | awk '{print $fieldno}' | while read i
do
echo line: $i
done
If the fields don't matter, but you want each line processed in the
loop do:
ls -l | while read i
do
echo line:$i
done
Bash doesn't need patching for this.
Again, the evaluation of the quotes is biting you. When the $lines
parameter is evaluated, the quotes are present.
bash is doing a double evaluation for for loop. It expects
a list of single tokens, rather than a list of quoted strings. This is
the same as in the bourne/korn shell.
If you want such elaborate handling of strings, I suggest using perl.
-
Jesse I Pollard, II
Email: [EMAIL PROTECTED]
Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Thursday 21 June 2001 10:02, Jesse Pollard wrote: Rob Landley [EMAIL PROTECTED]: On Wednesday 20 June 2001 17:20, Albert D. Cahalan wrote: Rob Landley writes: My only real gripe with Linux's threads right now [...] is that ps and top and such aren't thread aware and don't group them right. I'm told they added some kind of threadgroup field to processes that allows top and ps and such to get the display right. I haven't noticed any upgrades, and haven't had time to go hunting myself. There was a threadgroup added just before the 2.4 release. Linus said he'd remove it if he didn't get comments on how useful it was, examples of usage, etc. So I figured I'd look at the code that weekend, but the patch was removed before then! Can we give him feedback now, asking him to put it back? Submit patches to me, under the LGPL please. The FSF isn't likely to care. What, did you think this was the GNU system or something? I've stopped even TRYING to patch bash. try a for loop calling echo $$, eery single process bash forks off has the PARENT'S value for $$, which is REALLY annoying if you spend an afternoon writing code not knowing that and then wonder why the various process's temp file sare stomping each other... Actually - you have an error there. $$ gets evaluated during the parse, not during execution of the subprocess. Well, that would explain it then. (So $$ isn't actually an environment variable, it just looks like one? I had my threads calling a seperate function...) To get what you are describing it is necessary to sh -c 'echo $$' to force the delay of evaluation. Except that this spawns a new child and gives me the PID of the child, which lives for maybe a tenth of a second... Maybe I could do something with eval...? Either way, I switched the project in question to python. That depends on what you are trying to do. A couple people emailed me and pointed out I had to set IFS=$'\n', (often after first saying there's no way to do that and then correcting themselves towards the end of the message...) My first attempt to do it was to pipe the data into a sub-function do read a line at a time and store the results in an array. (Seemed pretty straightforward.) Except there's no way to get that array back out of the function, and that IS documented at the end of the bash man page. As I said: Python. Are you trying to echo the entire ls -l? or are you trying to convert an ls -l into a single column based on a field extracted from ls -l. I was trying to convert the lines into an array I could then iterate through. If the fields don't matter, but you want each line processed in the loop do: ls -l | while read i do echo line:$i done Hmmm... If that avoids the need to export the array from one shell instance to another, maybe it'd work. (I'm not really doing echo, I was doing var[$i]=line, with some other misc stuff.) But I got it to work via IFS, and that's good enough for right now. If you want such elaborate handling of strings, I suggest using perl. I came to the same conclusion, but would like for more than one person to be able to work on the same piece of code, so it's python. (Or PHP if Carl's the one who gets around to porting it first. :) But in the meantime, the shell script is now working. Thanks everybody...) Back to talking about the kernel.:) Rob - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Idea: Patches-from-linus mailing list? (Was Re: Alan Cox quote? (was: Re: accounting for threads))
On Wednesday 20 June 2001 21:57, D. Stimits wrote: MySQL is just a sample. I mention it because it is quite easy to link a web server to. Imagine patch running on a large file that is a conglomeration of 50 small patches; it could easily summarize this, and storing it through MySQL adds a lot of increased web flexibility (such as searching and sorting). It is, however, just one example of a way to make patch become autodocumenting. Not so much patch as a wrapper around patch. That's a good idea. A small perl script would do it... Right now, Linus makes a big file by appending mail messages to it. His mailer is, in theory, putting mail headers at the start of each of these messages (from, to, subject, and all that). At the end of the day, he feeds that big file to patch and it applies all the patches he's read through and decided he likes. It should be fairly easy to make a wrapper around patch that splits out a single mail message, feeds it to patch, and on some measurement of success forwards it to an otherwise read-only mailing list. (Which can then have a database based archiver subscribed to that list, if necessary.) If Linus used such a beast, we could get the actual mail messages Linus is applying patches from, as they're applied. Including any human readable documentation in them that patch itself would discard. No more asking did patch such and such get applied, when, who was it from... And we get the extra patch granularity Linus himself is so keen on. Instead of waiting for our weekly pre2-pre3 100k patch, we could follow the individual ones as logically grouped changes, with subject lines saying what the patch is about and everything. And the people hankering to make a CVS tree out of LInux kernel development would then have a much better checkin granularity to work with. :) The main thing, though, is that done right, it's no extra burden on Linus. (Which is kind of important if we ever hope to get him to use it. :) Sound like an idea to anybody else? Rob - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
In message [EMAIL PROTECTED] you wri te: In practice it's a BS. There is a lot of ways minor modifications of code could add a preemption point, so if you rely on the lack of such - expect major PITA. Yes, in theory SMP adds some extra fun. Practically, almost every SMP race found so far did not require SMP. Disagree. A significant percentage of the netfilter bugs have been SMP only (the whole thing is non-reentrant on UP). Depends what part of the kernel you play in, Rusty. -- Premature optmztion is rt of all evl. --DK - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Thu, 21 Jun 2001, Rusty Russell wrote: Disagree. A significant percentage of the netfilter bugs have been SMP only (the whole thing is non-reentrant on UP). I really doubt it. looking through the thing raised brows Well, if you use GFP_ATOMIC for everything... grep... Erm... AFAICS, you call create_chain() with interrupts disabled (under write_lock_irq_save()). Unless I'm _very_ mistaken, kmalloc(..., GFP_KERNEL) is a Bad Thing(tm) in that situation. And create_chain() leads to it. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Thu, 21 Jun 2001, Alexander Viro wrote: On Thu, 21 Jun 2001, Rusty Russell wrote: Disagree. A significant percentage of the netfilter bugs have been SMP only (the whole thing is non-reentrant on UP). I really doubt it. looking through the thing raised brows Well, if you use GFP_ATOMIC for everything... grep... Erm... AFAICS, you call create_chain() with interrupts disabled (under write_lock_irq_save()). Unless I'm _very_ mistaken, kmalloc(..., GFP_KERNEL) is a Bad Thing(tm) in that situation. And create_chain() leads to it. BTW, proc_net_create() is also not a good idea if you block the interrupts. Ditto for netlink_kernel_create(), AFAICS (due to netlink_kernel_creat() - sock_alloc() - get_empty_inode() - kmem_cache_alloc() with SLAB_KERNEL). That, BTW, is a nice illustration - it's easy to get a preemption point without noticing, so holding spinlocks, let alone disabling interrupts over the large area is going to hurt like hell. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Rok papez [EMAIL PROTECTED] wrote: On Tuesday 19 June 2001 18:09, Larry McVoy wrote: If you think you need threads then your processes are too fat ``Think of it this way: threads are like salt, not like pasta You like salt, I like salt, we all like salt. But we eat more pasta.'' Here are more from the same basket you obviously got the first quote from: [SNIP] Protected memory is a constant 10% CPU hog needed only by undisciplined programmers who can't keep their memory writes in their own process space. Now that's the primary reason to avoid threads, IMO. People spent years trying to get protected memory; the only real difference between threads and processes is that you throw protected memory out the window. Yes, the pthread API sucks; yes, there are times when threads are appropriate. If only: 1) Microsoft, Sun, and others didn't have such abysmal context switch numbers that people view threads vs. processes as a performance argument; 2) MS didn't conflate fork() and exec() into one CreateProcess() call; and 3) Java and others exposed rational event-driven APIs that didn't require multiple contexts of execution in weird places (1.4 is finally fixing this one anyway) then people might be able to really see that: 1) You usually don't need that many contexts of execution; and 2) Processes should be considered the primary COEs and threads only used when you really need to share almost all of your memory. That's aside from all the arguments against multithreading just based on elegance and code correctness POVs. Even if writing multithreaded code is marginally easier than writing poll()-based code, actually debugging it is a royal PITA. Coroutines (which aren't Alan's, Knuth had them long before Alan and even he was just rehashing old news) and state machines really are better in many cases. About the only criticism I have of Alan's statement that threads are for people who can't program state machines is the implication that threads are an easier API to get right. They aren't. They seem that way, but by tossing protected memory and introducing multiple COEs you get into a whole world of non-obvious problems that are very difficult to debug and often nearly impossible to reproduce in non-production environments. The salt quote I like; it allows for the sparing use of threads, but warns against their over(ab)use. Ah, well. Sumner __ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail http://personal.mail.yahoo.com/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On 20 Jun 2001 10:14:48 +0100, Alan Cox wrote: It does. ... not They are always readable. That's not very useful. Not in the sense of supporting aync, non-blocking i/o to disk files without using threads. -- Michael Rothwell [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: Why use threads ( was: Alan Cox quote?)
> On Wed, Jun 20, 2001 at 03:18:58PM -0700, David Schwartz wrote: > > As I said, you don't want to use one thread for each > > client. You use, say, > > 10 threads for the 16,000 clients. That way, if an occasional client > > ambushes a thread (say by reading a file off an NFS server or > > by using some > > infrequently used code that was swapped to a busy disk), your > > server will > > keep on humming. > This same approach can easily be used by multiple processes. Theoretically, pretty much everything you can do with a thread pool architecture can be done with a process pool architecture. Sharing file descriptors can be much harder. Some thread architectures (notably pthreads) go out of their way to make some things hard for thread pools (like try implementing Samba, since all the threads share there uids). But this is more about bad threading specifications and implementations than threading itself. > I don't see what is gained by using threads over processes for such an > architecture. I'll flip this back at you and ask you what's gained by using processes over threads. It's certainly more difficult to implement a process pool architecture than a thread pool architecture. Theoretically, performance for a thread pool architecture will be slightly better (on some architectures at least) due to the identical vm views. With a process pool architecture, you might have more control over what you share. But this doesn't actually gain you anything because as long as you share more than a small amount, you still have to consider your entire environment tainted if one execution vehicle goes out of control. I would be very interested in seeing, for example, a web server based on a 'process pool' design. Does anybody know of any? DS - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wednesday 20 June 2001 20:42, D. Stimits wrote: > Rob Landley wrote: > ...snip... > > > The patches-linus-actuall-applies mailing list idea is based on how Linus > > says he works: he appends patches he likes to a file and then calls patch > > -p1 < thatfile after a mail reading session. It wouldn't be too much > > work for somebody to write a toy he could use that lets him work about > > the same way but forwards the messages to another folder where they can > > go out on an otherwise read-only list. (No extra work for Linus. This > > is EXTREMELY important, 'cause otherwise he'll never touch it.) > > What if the file doing patches from is actually visible on a web page? > Or better yet, if the patch command itself was modified such that at the > same time it applies a patch, the source and the results were added to a > MySQL server which in turn shows as a web page? His patch file already has a bunch of patches glorped together. In theory they could be separated again by parsing the mail headers. I suppose that would be less work for Linus... The point is, the difference between the patches WE get and the patches LINUS gets is the granularity. He's constantly extorting other people to watch the granularity of what they send him (small patches, each doing one thing, with good documentation as to what they do and why, in seperate messages), but what WE get is a great big diff about once a week. So what I'm trying to figure out is if we can impose on Linus to cc: a mailing list on the stream of individual patch messages he's applying to his tree, so we can follow it better. And he IS willing to do a LITTLE work for us. He's issuing changelogs now. This would be significantly less effort than that... MySQL is overkill, and since these things started as mail messages why should they be converted into a foriegn format? There's threaded archivers for mailing lists and newsgroups and stuff already, why reinvent the wheel? Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wednesday 20 June 2001 11:33, Alexander Viro wrote: > On 20 Jun 2001, Jes Sorensen wrote: > > Not to mention how complex it is to get locking right in an efficient > > manner. Programming threads is not that much different from kernel SMP > > programming, except that in userland you get a core dump and retry, in > > the kernel you get an OOPS and an fsck and retry. > > Arrgh. As long as we have that "SMP makes locking harder" myth floating > around we _will_ get problems. Kernel UP programming is not different > from SMP one. It is multithreaded. And amount of genuine SMP bugs is > very small compared to ones that had been there on UP since way back. > And yes, programming threads is the same thing. No arguments here. Hopefully in 2.5 we'll get the pre-emptive UP patch in that enables the SMP locks on UP and finally clean it all out by exposing the bugs to the main user base. As for multi-threaded programming being hard, people are unfamiliar with it. Any programming is hard the first time. And almost anybody's first attempt at it is going to suck. (Dig out the linux kernel 0.02 sources sometime and compare them with what we have today.) The more experience you get with it, the better you are. Encouraging people to stay away from it rather than learn to do it RIGHT is the wrong attitude. People will figure out that using 1000 threads when you need 3 isn't the best way to go, that locking is an expense but failing to lock is worse, how to spot race conditions (the same old "security" mindset except you don't need a malicious third party to get bitten by it.) And they'll learn it the way I did, and the way everybody does. Do it wrong repeatedly. Make every mistake there is, hard. Get burned, rewrite it from scratch four times until you figure out how to design it right, spend long weekends looking for subtle little EVIL bugs you can't reproduce but which bite you the instant you stop looking for them, learn to play "hot potato" with volatile data you have to manipulate atomically... Everybody starts as a bad programmer. Some of us get it out of our systems when we're 12. Others decide programming is lucrative when they're 35 and inflict their "hello world" opus upon their coworkers. Me, I wrote a disk editor and a bunch of bbses in 5th and 6th grade back on the C64 that will never see the light of day. And yes they sucked. But I'm still proud of them. And I wrote awful multithreaded code back on OS/2, and can now write decent threading code because I've learned a large number of things to avoid doing. And I take proper care because I know how hard it is to FIND one of these if you do wind up doing it. I've done it. Once for two consecutive weeks on the same
Re: Alan Cox quote? (was: Re: accounting for threads)
Rob Landley wrote: ...snip... > The patches-linus-actuall-applies mailing list idea is based on how Linus > says he works: he appends patches he likes to a file and then calls patch -p1 > < thatfile after a mail reading session. It wouldn't be too much work for > somebody to write a toy he could use that lets him work about the same way > but forwards the messages to another folder where they can go out on an > otherwise read-only list. (No extra work for Linus. This is EXTREMELY > important, 'cause otherwise he'll never touch it.) What if the file doing patches from is actually visible on a web page? Or better yet, if the patch command itself was modified such that at the same time it applies a patch, the source and the results were added to a MySQL server which in turn shows as a web page? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
export IFS=$'\n'
> lines=`ls -l | awk '{print "\""$0"\""}'`
> for i in $lines
> do
> echo line:$i
> done
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wednesday 20 June 2001 17:20, Albert D. Cahalan wrote:
> Rob Landley writes:
> > My only real gripe with Linux's threads right now [...] is
> > that ps and top and such aren't thread aware and don't group them
> > right.
> >
> > I'm told they added some kind of "threadgroup" field to processes
> > that allows top and ps and such to get the display right. I haven't
> > noticed any upgrades, and haven't had time to go hunting myself.
>
> There was a "threadgroup" added just before the 2.4 release.
> Linus said he'd remove it if he didn't get comments on how
> useful it was, examples of usage, etc. So I figured I'd look at
> the code that weekend, but the patch was removed before then!
Can we give him feedback now, asking him to put it back?
> Submit patches to me, under the LGPL please. The FSF isn't likely
> to care. What, did you think this was the GNU system or something?
I've stopped even TRYING to patch bash. try a for loop calling "echo $$&",
eery single process bash forks off has the PARENT'S value for $$, which is
REALLY annoying if you spend an afternoon writing code not knowing that and
then wonder why the various process's temp file sare stomping each other...
Oh, and anybody who can explain this is welcome to try:
lines=`ls -l | awk '{print "\""$0"\""}'`
for i in $lines
do
echo line:$i
done
> How about a filesystem filter to spit out patches, or a filesystem
> interface to version control?
Explain please?
The patches-linus-actuall-applies mailing list idea is based on how Linus
says he works: he appends patches he likes to a file and then calls patch -p1
< thatfile after a mail reading session. It wouldn't be too much work for
somebody to write a toy he could use that lets him work about the same way
but forwards the messages to another folder where they can go out on an
otherwise read-only list. (No extra work for Linus. This is EXTREMELY
important, 'cause otherwise he'll never touch it.)
The advantage of this way is:
1) We know who sent the patches. (We get the message with the "from" headers
intact.)
2) Patch mails have descriptions in them most of the time, at least saying
why, if not what they do.
3) This way, we know (more or less in realtime) that Linus has gotten a patch
and applied it to his tree. (What version and everything.) It may be backed
out again later, but we could give him another tool that can do that and
notify the list...
This way, no mucking about with version control, no extra work for Linus, and
in fact he doesn't have to worry about keeping track of what patches he's
applied and when because he has a place he can go check if he forgets.
Now everybody tell me why this won't work. (Sure, all at once, why not...)
Rob
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Why use threads ( was: Alan Cox quote?)
On Wed, Jun 20, 2001 at 03:18:58PM -0700, David Schwartz wrote:
> As I said, you don't want to use one thread for each client. You use, say,
> 10 threads for the 16,000 clients. That way, if an occasional client
> ambushes a thread (say by reading a file off an NFS server or by using some
> infrequently used code that was swapped to a busy disk), your server will
> keep on humming.
This same approach can easily be used by multiple processes.
I don't see what is gained by using threads over processes for such an
architecture.
mrc
--
Mike Castle [EMAIL PROTECTED] www.netcom.com/~dalgoda/
We are all of us living in the shadow of Manhattan. -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Rob Landley writes: > My only real gripe with Linux's threads right now [...] is > that ps and top and such aren't thread aware and don't group them > right. > > I'm told they added some kind of "threadgroup" field to processes > that allows top and ps and such to get the display right. I haven't > noticed any upgrades, and haven't had time to go hunting myself. There was a "threadgroup" added just before the 2.4 release. Linus said he'd remove it if he didn't get comments on how useful it was, examples of usage, etc. So I figured I'd look at the code that weekend, but the patch was removed before then! There is nothing that ps and top can do about this problem. I've certainly looked into the matter; much of the code is mine. BTW, the version in debian-unstable is the most stable. :-) These options might help a little bit: --forest -H f > (Ever tried to sumit a patch to the FSF? They want you to sign > legal documents. That's annoying. I usually just send the bug > reports to red hat and let THEM deal with it...) Submit patches to me, under the LGPL please. The FSF isn't likely to care. What, did you think this was the GNU system or something? > Linus's job is to keep code OUT of the kernel. He has veto power, > nothing else. I suspect he's pre-emptively vetoing some stuff to > keep the flood down to a level he can deal with. Maybe someday > we'll convince him to use some variant of source control (not > necessarily CVS, how about just a seperate mailing list of the > individual patches as he applies them? One linus can post to and > that is read-only to everybody else? HE always wants patches > seperated down nicely into individual messages with explanations, > but WE have to get pre2-pre3 as one big patch lump. With a > patches-from-linus mailing list that he forwarded posts to, we'd > know exactly when a patch went in and who it was from without > bothering Linus. :) How about a filesystem filter to spit out patches, or a filesystem interface to version control? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE:Why use threads ( was: Alan Cox quote?)
> Nobody is arguing that having more than one thread of execution in an > application is a bad idea. On an SMP machine, having the same number of > processes/threads as there are CPUs is a requirement to get the scaling > if that app is all you are running. That's fine. But on a uniprocessor, > threads are basically stupid. The only place that I know of where it is > necessary is for I/O which blocks. And even there you only need enough > to overlap the I/O such that it streams. And processes will, and do, > work fine for that. It's very hard to use processes for this purpose. Consider, for example, a web server. You don't want to use one process for each client because that would limit your scalability (16,000 clients would become difficult, and with threads it's trivial). You don't want to use one thread for each client for obvious reasons. The risk with a poll loop type single-process design is that one client might perform an expensive operation and you can't afford to have your whole server stall. A worst-case kind of example would be reading a file from a slow NFS server. But more common are page faults from slow disks when a piece of code in the web server that handles some obscure feature needs to fault in. This can theoretically be handled by a process pool architecture with suitable shared memory, but that's much more difficult to get right than threads. And there's no advantage gained from the extra effort. Another case where threads can be extremely useful is for special tasks with timing requirements. Consider, for example, time and timer management. Many programs have requirements for a monotonic timer that can provide reasonable guarantee that intervals will be accurately measured so that future events will trigger at the right time. For example, if you need to idle out a connection if it's not used for, say, 30 seconds it may be unacceptable to have all your connections stay around for an hour if the clock jumps backwards. This is very easy to do if you have a thread monitor the clock and wake up every second. If the clock jumps forward, it can let virtual time run a bit faster until it catches up. If it jumps backwards, it can slow virtual time down keeping it always monotonic. This can provide a reasonable guarantee that no matter what the system time does (short of jumping every second!) your 30 second timer will be between, say, 25 and 35 seconds. This can also provide a good way to measure elapsed time that is well-protected from system clock issues. Without a separate thread, it's very hard to assure that the the clock monitoring code runs often enough to keep its timebase. If it doesn't run for 6 seconds, it would think the time had jumped. As a separate thread, you can write this time monitoring timer management code one time and it will work with any other code regardless of how it blocks. The two things I have found threads most useful for (in fact, indispensable for) are ambush avoidance and dedicated threads for 'special' tasks that can't easily be done another way. Both of these things can, at least theoertically, by done using processes and shared memory or other IPC mechanisms, but threads is easier and cleaner. This is especially true for ambush avoidance. DS - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Don't forget the linux-kernel favorite, "Debuggers are for bad programmers". } Here are more from the same basket you obviously got the first quote from: } } } Virtual memory is only for unskilled programmers who don't know how to use } overlays. } } Protected memory is a constant 10% CPU hog needed only by undisciplined } programmers who can't keep their memory writes in their own process space. } - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Hi! It's hard not to reply to this kind of message but there is so much "anti-thread hype" here that someone obviously has to stand up to it. This reply isn't aimed just at Larry but at all the anti-thread-rant people with 0 threads == 0 problems attitude. On Tuesday 19 June 2001 18:09, Larry McVoy wrote: > "If you think you need threads then your processes are too fat" > ``Think of it this way: threads are like salt, not like pasta. You > like salt, I like salt, we all like salt. But we eat more pasta.'' Here are more from the same basket you obviously got the first quote from: Virtual memory is only for unskilled programmers who don't know how to use overlays. Protected memory is a constant 10% CPU hog needed only by undisciplined programmers who can't keep their memory writes in their own process space. > Threads are a really bad idea. I could *say* the same about Alans co-routines and Async IO :-). But it would be foolish of me to criticize something I haven't used. There is more than one way how to skin a dinosaur. And threads are one way of doing it. You don't like it ? FINE. But don't go bashing it. Probably most people bash threads becouse of a silly way POSIX designed (designed is an overstatement) their pthread API and becouse UNIX was around before threads thus probably making every UNIX thread support a hacked and not a designed tool. Other OSes have certainly proven threads to be nice and usable. And here is another one for you quotes page: If you can't stick your head out of your own backyard please... don't go and crtiticize the world... :-) -- best regards, Rok Pape. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wed, 20 Jun 2001, george anzinger wrote: > > around we _will_ get problems. Kernel UP programming is not different > > from SMP one. It is multithreaded. And amount of genuine SMP bugs is > > very small compared to ones that had been there on UP since way back. > > And yes, programming threads is the same thing. No arguments here. > > > Correct, IF the UP kernel is preemptable. As long as it is not (and SMP > is ignored) threads are harder BECAUSE they are preemptable. In practice it's a BS. There is a lot of ways minor modifications of code could add a preemption point, so if you rely on the lack of such - expect major PITA. Yes, in theory SMP adds some extra fun. Practically, almost every "SMP" race found so far did not require SMP. Clean code is trivial to make SMP-safe - critical areas that rely on lack of preemption are couple of instructions wide and are easy to protect. Anything trickier and I bet that you have a race on (normal) UP kernel. Been there, found probably several hundreds of them. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wed, Jun 20, 2001 at 08:21:30PM +1000, john slee wrote: > On Tue, Jun 19, 2001 at 08:04:42PM -0700, Larry McVoy wrote: > > I asked Linus for this a long time ago and he pointed out that you couldn't > > make it work over NFS, at least not nicely. It does seem like that could > > be worked around by having a "poll daemon" which knew about all the things > > being waited on and checked them. Or something. > > could sgi's imon+fam work help a little here (with the "poll daemon" part)? > am i on the wrong train ? :-] I was never a fan of that tool though SGI people loved it. I think that someone needs to grab this area and think it through. There ought to be some way to do this that isn't gross. But it will take some thinking. If you could convince one of the file system guys to think about this for a week I suspect you'd get something nice. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wednesday 20 June 2001 10:35, Mike Porter wrote: > > But that foregoes the point that the code is far more complex and harder > > to make 'obviously correct', a concept that *does* translate well to > > userspace. > > One point is that 'obviously correct' is much harder to 'prove' for > threads (or processes with shared memory) than you might think. > With a state machine, you can 'prove' that object accesses won't > conflict much more easily. With a threaded or process based model, > you have to spend considerable time thinking about multiple readers > and writers and locking. > > One metric I use to evaluate program complexity is how big of a > headache I get when trying to prove something "correct". > Multi-process or multi-threaded code hurts more than a well written > state machine. The same applies to security though. There's programmers out there we're unwilling to give the tools to create race conditions, but we expect them to write stuff that won't allow a box on the internet to be own3d in under 24 hours... Obvious isn't always correct... Rob - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tuesday 19 June 2001 19:31, Timur Tabi wrote: > Amen. This is one of the reasons why I also prefer OS/2 over Linux. Preferred. OS/2's day has come and gone. IBM killed it with a stupid diversion into the power PC version between 1993 and 1995. By the time Windows 95 was released, MS had finally (after 11 years) managed to properly clone the original 1984 macintosh, and 90% of the PC user base was no longer actively looking to replace their OS (Windows 3.1 and/or DOS). The window of opportunity for a proprietary OS to replace Microsoft's had closed, and OS/2 4.0 was just plain too late. (If it had been released two years earlier, things might have been different. But it wasn't.) But the window to commoditize a propreitary niche never closes. The PC clones didn't care whether digital's minicomputers or IBM's mainframes won out in the marketplace, they were quite happy to replace both. Linux is winning because it's free software, not for whatever transient technical reasons make one version of an OS better than another on a given day. > Feature Installer is a bad example. That software is a piece of crap for > lots of reasons, excessive threading being only one, and every OS/2 user > knew it the day it was released. Why do you think WarpIN was created? I know, and I'm sorry I rescued FI from the corpse of OS/2 for the Power PC. But it was my job, you know. :) (And I did rewrite half of it from scratch. Before that it didn't work at ALL. There were algorithms in there that scaled (failed to scale) to n^n complexity on the object hierarchy. Unfortunately, IBM wouldn't let me rewrite the other half of it, and most of that was pretty darn obnoxious and evil.) Isn't it interesting that OS/2 turned into a unix platform? (EMX and gcc, where all the software was coming from? I think I tried to point that out to you when I attempted to recruit you into the LInux world at that gaming session at armadillocon in... 1999? Glad to see you did eventually come over to the chocolate side of the force after all... :) > Not quite. What makes OS/2's threads superior is that the OS multitasks > threads, not processes. So I can create a time-critical thread in my > process, and it will have priority over ALL threads in ALL processes. And in Linux you can create a time-critical process that shares large gorps of memory where you keep your global variables, and call it a thread, and it works the same way. My only real gripe with Linux's threads right now (other than the fact they keep trying to get us to use the posix api, which sucks. What IS an event variable. What's wrong with event semaphores?) is that ps and top and such aren't thread aware and don't group them right. I'm told they added some kind of "threadgroup" field to processes that allows top and ps and such to get the display right. I haven't noticed any upgrades, and haven't had time to go hunting myself. (Ever tried to sumit a patch to the FSF? They want you to sign legal documents. That's annoying. I usually just send the bug reports to red hat and let THEM deal with it...) > A lot of OS/2 software is written with this feature in mind. I know of one > programmer who absolutely hates Linux because it's just too difficult > porting software to it, and the lack of decent thread support is part of > the problem. Yup. OS/2 is the largest nest of trained, experienced multi-threaded programmers. (And it takes a LOT of training experience to do threads right.) That's why I've been trying to recruit ex-OS/2 guys over to Linux for years now. (Most followed Java to Linux after Netscape opened up Mozilla, but there used to be several notable holdouts...) Threading is just another way to look at programming, with both advantages and disadvantages. You get automatic SMP scalability that's quite nice if you keep cache coherency in mind. You get great latency and responsiveness in user interfaces with just one or two extra threads. (State machines may be superior if implemented perfectly, but like co-operative multitasking (which they are) it's trivially easy to block the suckers and hang the whole GUI. If you hang one thread, you can even program another thread to notice automatically and unwedge it without much overhead at all. Trying to unwedge a state machine is a little more complicated than kill/respawn of a thread. Of course a perfect state machine shouldn't have those problems, but when you interface with other people's code in a large system, you accept imperfection and make the system survive as best it can.) > Exactly. Saying that threads cause bad code is just as dumb as saying that > a kernel debugger will cause bad code because programmers will start using > the debugger instead of proper design. > > Oh wait, never mind . Ah, don't pick on Linus. It's not exactly like the limiting factor to kernel development is a SHORTAGE of patches sent in to the various
Re: Alan Cox quote? (was: Re: accounting for threads)
> "Al" == Alexander Viro <[EMAIL PROTECTED]> writes: Al> On 20 Jun 2001, Jes Sorensen wrote: >> Not to mention how complex it is to get locking right in an >> efficient manner. Programming threads is not that much different >> from kernel SMP programming, except that in userland you get a core >> dump and retry, in the kernel you get an OOPS and an fsck and >> retry. Al> Arrgh. As long as we have that "SMP makes locking harder" myth Al> floating around we _will_ get problems. Kernel UP programming is Al> not different from SMP one. It is multithreaded. And amount of Al> genuine SMP bugs is very small compared to ones that had been Al> there on UP since way back. And yes, programming threads is the Al> same thing. No arguments here. Call it SMP or kernel threading, I don't really care, it's the same thing. My point is that in the kernel you must take threading/SMP into account when coding and yes it's not trivial to do it efficiently (though often fairly easy to do it inefficiently) and the same applies to userland threads. Userland threads are just not some chest of gold that just opens up a free path to paradise as most CS teachers seems to promote it as being. Jes - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
> But that foregoes the point that the code is far more complex and harder to > make 'obviously correct', a concept that *does* translate well to userspace. One point is that 'obviously correct' is much harder to 'prove' for threads (or processes with shared memory) than you might think. With a state machine, you can 'prove' that object accesses won't conflict much more easily. With a threaded or process based model, you have to spend considerable time thinking about multiple readers and writers and locking. One metric I use to evaluate program complexity is how big of a headache I get when trying to prove something "correct". Multi-process or multi-threaded code hurts more than a well written state machine. Mike - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
> "Alan" == Alan Cox <[EMAIL PROTECTED]> writes: >> But that foregoes the point that the code is far more complex and >> harder to make 'obviously correct', a concept that *does* translate >> well to userspace. Alan> There I disagree. Threads introduce parallelism that the Alan> majority of user space programmers have trouble getting right Alan> (not that C is helpful here). Alan> A threaded program has a set of extremely complex hard to repeat Alan> timing based behaviour dependancies. An unthreaded app almost Alan> always does the same thing on the same input. From a Alan> verification and coverage point of view that is incredibly Alan> important. Not to mention how complex it is to get locking right in an efficient manner. Programming threads is not that much different from kernel SMP programming, except that in userland you get a core dump and retry, in the kernel you get an OOPS and an fsck and retry. For some reason CS professors decided that threads were the cool thing to do and started teaching all their students to program threadded applications. Unfortunately they did forget to teach about the caveats and all the funny side effects that generally mostly threadded apps a really stupid idea. It's been like this for some years now, one would expect a switch to another buzzword of the year paradigm from them soon. Jes - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
At 18:31 -0500 2001-06-19, Timur Tabi wrote: >Not quite. What makes OS/2's threads superior is that the OS multitasks >threads, not processes. So I can create a time-critical thread in my process, >and it will have priority over ALL threads in ALL processes. In contrast to Linux, which does exactly the same thing -- so what are you trying to tell us? >A lot of OS/2 software is written with this feature in mind. I know of one >programmer who absolutely hates Linux because it's just too difficult porting >software to it, and the lack of decent thread support is part of the problem. So what would you consider to be "decent thread support"? -- Matthias Urlichs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Threads FAQ entry incomplete (was: Alan Cox quote?)
At 03:12 PM 6/19/01 -0600, Richard Gooch wrote: >New FAQ entry: http://www.tux.org/lkml/#s7-21 > >Yeah, it's probably a bit harsh :-) It's also incomplete, in my view, to the point of being misleading. Part of the reason I'm reacting so harshly to this FAQ entry is that it flies in the face of my own experience. I do a lot of real-time and near-real-time programming, and find that the process and thread models are essential to the proper and expected operation of my code. The FAQ answer neglects this entire area of applications. For details: I run a process (not threads as yet, but it will happen) for each event I need to react to in near real time. The running priority of these processes are set as high as possible so that when the event occurs the newly-unblocked process is guaranteed to run almost immediately. These processes, when unblocked, do the absolute minimum amount of work to meet the real-time requirements and then block again. If an event requires extended processing to begin but not to complete in real time, the event process sends an appropriate signal to the main process to perform that extended processing. The main problem with the comment in the FAQ entry that "it's easy to write an event callback system" is that the response time may not meet the needs of the application, particularly in computer systems that have multiple near-real-time applications running at the same time. The key design point in this near-real-time model I have described is that it does not violate the rule of thumb described in the FAQ answer, that the Linux kernel is designed to work well with a small number of RUNNING processes/threads. Ideally, a high-priority process should run for much, much less time than the standard time-slice so as to minimize the number of scheduler elements marked as "ready to run." My standard is for a high-priority process driven by an event to run for no more than a millisecond or so, and leave any heavy lifting to the main process to take care of when the main process gets around to it. For true real-time processing, you have to write device drivers to react to external events directly, a programming task that is more burdensome and error-prone. Or, better, use a real RTOS instead of Linux. When the real-time requirement isn't so critical, though, the method I have described allows the whole thing to be done in userland, where it can be debugged far more easily. In summary, writing code to use many processes or threads is NOT lazy programming in all instances, as the FAQ answer implies. Stephen Satchell writing time-critical programs for only a short time -- 30 years. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tue, Jun 19, 2001 at 08:04:42PM -0700, Larry McVoy wrote: [ ... ] > I asked Linus for this a long time ago and he pointed out that you couldn't > make it work over NFS, at least not nicely. It does seem like that could > be worked around by having a "poll daemon" which knew about all the things > being waited on and checked them. Or something. could sgi's imon+fam work help a little here (with the "poll daemon" part)? am i on the wrong train ? :-] not sure how maintained it is these days... [ ... ] j. -- "Bobby, jiggle Grandpa's rat so it looks alive, please" -- gary larson - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
> Oh, how I would love for select() and poll() to work on files... or for > any other working AIO mothods to be present. > What would get broken if things were changed to let select() work for > filesystem fds? It does. They are always readable. If it were to trigger prefetching in that case then I suspect no harm would occur - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tue, Jun 19, 2001 at 08:04:42PM -0700, Larry McVoy wrote: [ ... ] I asked Linus for this a long time ago and he pointed out that you couldn't make it work over NFS, at least not nicely. It does seem like that could be worked around by having a poll daemon which knew about all the things being waited on and checked them. Or something. could sgi's imon+fam work help a little here (with the poll daemon part)? am i on the wrong train ? :-] not sure how maintained it is these days... [ ... ] j. -- Bobby, jiggle Grandpa's rat so it looks alive, please -- gary larson - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Threads FAQ entry incomplete (was: Alan Cox quote?)
At 03:12 PM 6/19/01 -0600, Richard Gooch wrote: New FAQ entry: http://www.tux.org/lkml/#s7-21 Yeah, it's probably a bit harsh :-) It's also incomplete, in my view, to the point of being misleading. Part of the reason I'm reacting so harshly to this FAQ entry is that it flies in the face of my own experience. I do a lot of real-time and near-real-time programming, and find that the process and thread models are essential to the proper and expected operation of my code. The FAQ answer neglects this entire area of applications. For details: I run a process (not threads as yet, but it will happen) for each event I need to react to in near real time. The running priority of these processes are set as high as possible so that when the event occurs the newly-unblocked process is guaranteed to run almost immediately. These processes, when unblocked, do the absolute minimum amount of work to meet the real-time requirements and then block again. If an event requires extended processing to begin but not to complete in real time, the event process sends an appropriate signal to the main process to perform that extended processing. The main problem with the comment in the FAQ entry that it's easy to write an event callback system is that the response time may not meet the needs of the application, particularly in computer systems that have multiple near-real-time applications running at the same time. The key design point in this near-real-time model I have described is that it does not violate the rule of thumb described in the FAQ answer, that the Linux kernel is designed to work well with a small number of RUNNING processes/threads. Ideally, a high-priority process should run for much, much less time than the standard time-slice so as to minimize the number of scheduler elements marked as ready to run. My standard is for a high-priority process driven by an event to run for no more than a millisecond or so, and leave any heavy lifting to the main process to take care of when the main process gets around to it. For true real-time processing, you have to write device drivers to react to external events directly, a programming task that is more burdensome and error-prone. Or, better, use a real RTOS instead of Linux. When the real-time requirement isn't so critical, though, the method I have described allows the whole thing to be done in userland, where it can be debugged far more easily. In summary, writing code to use many processes or threads is NOT lazy programming in all instances, as the FAQ answer implies. Stephen Satchell writing time-critical programs for only a short time -- 30 years. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
At 18:31 -0500 2001-06-19, Timur Tabi wrote: Not quite. What makes OS/2's threads superior is that the OS multitasks threads, not processes. So I can create a time-critical thread in my process, and it will have priority over ALL threads in ALL processes. In contrast to Linux, which does exactly the same thing -- so what are you trying to tell us? A lot of OS/2 software is written with this feature in mind. I know of one programmer who absolutely hates Linux because it's just too difficult porting software to it, and the lack of decent thread support is part of the problem. So what would you consider to be decent thread support? -- Matthias Urlichs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
But that foregoes the point that the code is far more complex and harder to make 'obviously correct', a concept that *does* translate well to userspace. One point is that 'obviously correct' is much harder to 'prove' for threads (or processes with shared memory) than you might think. With a state machine, you can 'prove' that object accesses won't conflict much more easily. With a threaded or process based model, you have to spend considerable time thinking about multiple readers and writers and locking. One metric I use to evaluate program complexity is how big of a headache I get when trying to prove something correct. Multi-process or multi-threaded code hurts more than a well written state machine. Mike - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tuesday 19 June 2001 19:31, Timur Tabi wrote: Amen. This is one of the reasons why I also prefer OS/2 over Linux. Preferred. OS/2's day has come and gone. IBM killed it with a stupid diversion into the power PC version between 1993 and 1995. By the time Windows 95 was released, MS had finally (after 11 years) managed to properly clone the original 1984 macintosh, and 90% of the PC user base was no longer actively looking to replace their OS (Windows 3.1 and/or DOS). The window of opportunity for a proprietary OS to replace Microsoft's had closed, and OS/2 4.0 was just plain too late. (If it had been released two years earlier, things might have been different. But it wasn't.) But the window to commoditize a propreitary niche never closes. The PC clones didn't care whether digital's minicomputers or IBM's mainframes won out in the marketplace, they were quite happy to replace both. Linux is winning because it's free software, not for whatever transient technical reasons make one version of an OS better than another on a given day. Feature Installer is a bad example. That software is a piece of crap for lots of reasons, excessive threading being only one, and every OS/2 user knew it the day it was released. Why do you think WarpIN was created? I know, and I'm sorry I rescued FI from the corpse of OS/2 for the Power PC. But it was my job, you know. :) (And I did rewrite half of it from scratch. Before that it didn't work at ALL. There were algorithms in there that scaled (failed to scale) to n^n complexity on the object hierarchy. Unfortunately, IBM wouldn't let me rewrite the other half of it, and most of that was pretty darn obnoxious and evil.) Isn't it interesting that OS/2 turned into a unix platform? (EMX and gcc, where all the software was coming from? I think I tried to point that out to you when I attempted to recruit you into the LInux world at that gaming session at armadillocon in... 1999? Glad to see you did eventually come over to the chocolate side of the force after all... :) Not quite. What makes OS/2's threads superior is that the OS multitasks threads, not processes. So I can create a time-critical thread in my process, and it will have priority over ALL threads in ALL processes. And in Linux you can create a time-critical process that shares large gorps of memory where you keep your global variables, and call it a thread, and it works the same way. My only real gripe with Linux's threads right now (other than the fact they keep trying to get us to use the posix api, which sucks. What IS an event variable. What's wrong with event semaphores?) is that ps and top and such aren't thread aware and don't group them right. I'm told they added some kind of threadgroup field to processes that allows top and ps and such to get the display right. I haven't noticed any upgrades, and haven't had time to go hunting myself. (Ever tried to sumit a patch to the FSF? They want you to sign legal documents. That's annoying. I usually just send the bug reports to red hat and let THEM deal with it...) A lot of OS/2 software is written with this feature in mind. I know of one programmer who absolutely hates Linux because it's just too difficult porting software to it, and the lack of decent thread support is part of the problem. Yup. OS/2 is the largest nest of trained, experienced multi-threaded programmers. (And it takes a LOT of training experience to do threads right.) That's why I've been trying to recruit ex-OS/2 guys over to Linux for years now. (Most followed Java to Linux after Netscape opened up Mozilla, but there used to be several notable holdouts...) Threading is just another way to look at programming, with both advantages and disadvantages. You get automatic SMP scalability that's quite nice if you keep cache coherency in mind. You get great latency and responsiveness in user interfaces with just one or two extra threads. (State machines may be superior if implemented perfectly, but like co-operative multitasking (which they are) it's trivially easy to block the suckers and hang the whole GUI. If you hang one thread, you can even program another thread to notice automatically and unwedge it without much overhead at all. Trying to unwedge a state machine is a little more complicated than kill/respawn of a thread. Of course a perfect state machine shouldn't have those problems, but when you interface with other people's code in a large system, you accept imperfection and make the system survive as best it can.) Exactly. Saying that threads cause bad code is just as dumb as saying that a kernel debugger will cause bad code because programmers will start using the debugger instead of proper design. Oh wait, never mind . Ah, don't pick on Linus. It's not exactly like the limiting factor to kernel development is a SHORTAGE of patches sent in to the various maintainers. Linus's
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wednesday 20 June 2001 10:35, Mike Porter wrote: But that foregoes the point that the code is far more complex and harder to make 'obviously correct', a concept that *does* translate well to userspace. One point is that 'obviously correct' is much harder to 'prove' for threads (or processes with shared memory) than you might think. With a state machine, you can 'prove' that object accesses won't conflict much more easily. With a threaded or process based model, you have to spend considerable time thinking about multiple readers and writers and locking. One metric I use to evaluate program complexity is how big of a headache I get when trying to prove something correct. Multi-process or multi-threaded code hurts more than a well written state machine. The same applies to security though. There's programmers out there we're unwilling to give the tools to create race conditions, but we expect them to write stuff that won't allow a box on the internet to be own3d in under 24 hours... Obvious isn't always correct... Rob - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wed, Jun 20, 2001 at 08:21:30PM +1000, john slee wrote: On Tue, Jun 19, 2001 at 08:04:42PM -0700, Larry McVoy wrote: I asked Linus for this a long time ago and he pointed out that you couldn't make it work over NFS, at least not nicely. It does seem like that could be worked around by having a poll daemon which knew about all the things being waited on and checked them. Or something. could sgi's imon+fam work help a little here (with the poll daemon part)? am i on the wrong train ? :-] I was never a fan of that tool though SGI people loved it. I think that someone needs to grab this area and think it through. There ought to be some way to do this that isn't gross. But it will take some thinking. If you could convince one of the file system guys to think about this for a week I suspect you'd get something nice. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wed, 20 Jun 2001, george anzinger wrote: around we _will_ get problems. Kernel UP programming is not different from SMP one. It is multithreaded. And amount of genuine SMP bugs is very small compared to ones that had been there on UP since way back. And yes, programming threads is the same thing. No arguments here. Correct, IF the UP kernel is preemptable. As long as it is not (and SMP is ignored) threads are harder BECAUSE they are preemptable. In practice it's a BS. There is a lot of ways minor modifications of code could add a preemption point, so if you rely on the lack of such - expect major PITA. Yes, in theory SMP adds some extra fun. Practically, almost every SMP race found so far did not require SMP. Clean code is trivial to make SMP-safe - critical areas that rely on lack of preemption are couple of instructions wide and are easy to protect. Anything trickier and I bet that you have a race on (normal) UP kernel. Been there, found probably several hundreds of them. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Hi! It's hard not to reply to this kind of message but there is so much "anti-thread hype" here that someone obviously has to stand up to it. This reply isn't aimed just at Larry but at all the anti-thread-rant people with 0 threads == 0 problems attitude. On Tuesday 19 June 2001 18:09, Larry McVoy wrote: "If you think you need threads then your processes are too fat" ``Think of it this way: threads are like salt, not like pasta. You like salt, I like salt, we all like salt. But we eat more pasta.'' Here are more from the same basket you obviously got the first quote from: Virtual memory is only for unskilled programmers who don't know how to use overlays. Protected memory is a constant 10% CPU hog needed only by undisciplined programmers who can't keep their memory writes in their own process space. Threads are a really bad idea. I could *say* the same about Alans co-routines and Async IO :-). But it would be foolish of me to criticize something I haven't used. There is more than one way how to skin a dinosaur. And threads are one way of doing it. You don't like it ? FINE. But don't go bashing it. Probably most people bash threads becouse of a silly way POSIX designed (designed is an overstatement) their pthread API and becouse UNIX was around before threads thus probably making every UNIX thread support a hacked and not a designed tool. Other OSes have certainly proven threads to be nice and usable. And here is another one for you quotes page: If you can't stick your head out of your own backyard please... don't go and crtiticize the world... :-) -- best regards, Rok Pape. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Don't forget the linux-kernel favorite, Debuggers are for bad programmers. } Here are more from the same basket you obviously got the first quote from: } } } Virtual memory is only for unskilled programmers who don't know how to use } overlays. } } Protected memory is a constant 10% CPU hog needed only by undisciplined } programmers who can't keep their memory writes in their own process space. } - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE:Why use threads ( was: Alan Cox quote?)
Nobody is arguing that having more than one thread of execution in an application is a bad idea. On an SMP machine, having the same number of processes/threads as there are CPUs is a requirement to get the scaling if that app is all you are running. That's fine. But on a uniprocessor, threads are basically stupid. The only place that I know of where it is necessary is for I/O which blocks. And even there you only need enough to overlap the I/O such that it streams. And processes will, and do, work fine for that. It's very hard to use processes for this purpose. Consider, for example, a web server. You don't want to use one process for each client because that would limit your scalability (16,000 clients would become difficult, and with threads it's trivial). You don't want to use one thread for each client for obvious reasons. The risk with a poll loop type single-process design is that one client might perform an expensive operation and you can't afford to have your whole server stall. A worst-case kind of example would be reading a file from a slow NFS server. But more common are page faults from slow disks when a piece of code in the web server that handles some obscure feature needs to fault in. This can theoretically be handled by a process pool architecture with suitable shared memory, but that's much more difficult to get right than threads. And there's no advantage gained from the extra effort. Another case where threads can be extremely useful is for special tasks with timing requirements. Consider, for example, time and timer management. Many programs have requirements for a monotonic timer that can provide reasonable guarantee that intervals will be accurately measured so that future events will trigger at the right time. For example, if you need to idle out a connection if it's not used for, say, 30 seconds it may be unacceptable to have all your connections stay around for an hour if the clock jumps backwards. This is very easy to do if you have a thread monitor the clock and wake up every second. If the clock jumps forward, it can let virtual time run a bit faster until it catches up. If it jumps backwards, it can slow virtual time down keeping it always monotonic. This can provide a reasonable guarantee that no matter what the system time does (short of jumping every second!) your 30 second timer will be between, say, 25 and 35 seconds. This can also provide a good way to measure elapsed time that is well-protected from system clock issues. Without a separate thread, it's very hard to assure that the the clock monitoring code runs often enough to keep its timebase. If it doesn't run for 6 seconds, it would think the time had jumped. As a separate thread, you can write this time monitoring timer management code one time and it will work with any other code regardless of how it blocks. The two things I have found threads most useful for (in fact, indispensable for) are ambush avoidance and dedicated threads for 'special' tasks that can't easily be done another way. Both of these things can, at least theoertically, by done using processes and shared memory or other IPC mechanisms, but threads is easier and cleaner. This is especially true for ambush avoidance. DS - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Rob Landley writes: My only real gripe with Linux's threads right now [...] is that ps and top and such aren't thread aware and don't group them right. I'm told they added some kind of threadgroup field to processes that allows top and ps and such to get the display right. I haven't noticed any upgrades, and haven't had time to go hunting myself. There was a threadgroup added just before the 2.4 release. Linus said he'd remove it if he didn't get comments on how useful it was, examples of usage, etc. So I figured I'd look at the code that weekend, but the patch was removed before then! There is nothing that ps and top can do about this problem. I've certainly looked into the matter; much of the code is mine. BTW, the version in debian-unstable is the most stable. :-) These options might help a little bit: --forest -H f (Ever tried to sumit a patch to the FSF? They want you to sign legal documents. That's annoying. I usually just send the bug reports to red hat and let THEM deal with it...) Submit patches to me, under the LGPL please. The FSF isn't likely to care. What, did you think this was the GNU system or something? Linus's job is to keep code OUT of the kernel. He has veto power, nothing else. I suspect he's pre-emptively vetoing some stuff to keep the flood down to a level he can deal with. Maybe someday we'll convince him to use some variant of source control (not necessarily CVS, how about just a seperate mailing list of the individual patches as he applies them? One linus can post to and that is read-only to everybody else? HE always wants patches seperated down nicely into individual messages with explanations, but WE have to get pre2-pre3 as one big patch lump. With a patches-from-linus mailing list that he forwarded posts to, we'd know exactly when a patch went in and who it was from without bothering Linus. :) How about a filesystem filter to spit out patches, or a filesystem interface to version control? - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Why use threads ( was: Alan Cox quote?)
On Wed, Jun 20, 2001 at 03:18:58PM -0700, David Schwartz wrote: As I said, you don't want to use one thread for each client. You use, say, 10 threads for the 16,000 clients. That way, if an occasional client ambushes a thread (say by reading a file off an NFS server or by using some infrequently used code that was swapped to a busy disk), your server will keep on humming. This same approach can easily be used by multiple processes. I don't see what is gained by using threads over processes for such an architecture. mrc -- Mike Castle [EMAIL PROTECTED] www.netcom.com/~dalgoda/ We are all of us living in the shadow of Manhattan. -- Watchmen fatal (You are in a maze of twisty compiler features, all different); -- gcc - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wednesday 20 June 2001 17:20, Albert D. Cahalan wrote:
Rob Landley writes:
My only real gripe with Linux's threads right now [...] is
that ps and top and such aren't thread aware and don't group them
right.
I'm told they added some kind of threadgroup field to processes
that allows top and ps and such to get the display right. I haven't
noticed any upgrades, and haven't had time to go hunting myself.
There was a threadgroup added just before the 2.4 release.
Linus said he'd remove it if he didn't get comments on how
useful it was, examples of usage, etc. So I figured I'd look at
the code that weekend, but the patch was removed before then!
Can we give him feedback now, asking him to put it back?
Submit patches to me, under the LGPL please. The FSF isn't likely
to care. What, did you think this was the GNU system or something?
I've stopped even TRYING to patch bash. try a for loop calling echo $$,
eery single process bash forks off has the PARENT'S value for $$, which is
REALLY annoying if you spend an afternoon writing code not knowing that and
then wonder why the various process's temp file sare stomping each other...
Oh, and anybody who can explain this is welcome to try:
lines=`ls -l | awk '{print \$0\}'`
for i in $lines
do
echo line:$i
done
How about a filesystem filter to spit out patches, or a filesystem
interface to version control?
Explain please?
The patches-linus-actuall-applies mailing list idea is based on how Linus
says he works: he appends patches he likes to a file and then calls patch -p1
thatfile after a mail reading session. It wouldn't be too much work for
somebody to write a toy he could use that lets him work about the same way
but forwards the messages to another folder where they can go out on an
otherwise read-only list. (No extra work for Linus. This is EXTREMELY
important, 'cause otherwise he'll never touch it.)
The advantage of this way is:
1) We know who sent the patches. (We get the message with the from headers
intact.)
2) Patch mails have descriptions in them most of the time, at least saying
why, if not what they do.
3) This way, we know (more or less in realtime) that Linus has gotten a patch
and applied it to his tree. (What version and everything.) It may be backed
out again later, but we could give him another tool that can do that and
notify the list...
This way, no mucking about with version control, no extra work for Linus, and
in fact he doesn't have to worry about keeping track of what patches he's
applied and when because he has a place he can go check if he forgets.
Now everybody tell me why this won't work. (Sure, all at once, why not...)
Rob
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
export IFS=$'\n'
lines=`ls -l | awk '{print \$0\}'`
for i in $lines
do
echo line:$i
done
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Rob Landley wrote: ...snip... The patches-linus-actuall-applies mailing list idea is based on how Linus says he works: he appends patches he likes to a file and then calls patch -p1 thatfile after a mail reading session. It wouldn't be too much work for somebody to write a toy he could use that lets him work about the same way but forwards the messages to another folder where they can go out on an otherwise read-only list. (No extra work for Linus. This is EXTREMELY important, 'cause otherwise he'll never touch it.) What if the file doing patches from is actually visible on a web page? Or better yet, if the patch command itself was modified such that at the same time it applies a patch, the source and the results were added to a MySQL server which in turn shows as a web page? - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wednesday 20 June 2001 20:42, D. Stimits wrote: Rob Landley wrote: ...snip... The patches-linus-actuall-applies mailing list idea is based on how Linus says he works: he appends patches he likes to a file and then calls patch -p1 thatfile after a mail reading session. It wouldn't be too much work for somebody to write a toy he could use that lets him work about the same way but forwards the messages to another folder where they can go out on an otherwise read-only list. (No extra work for Linus. This is EXTREMELY important, 'cause otherwise he'll never touch it.) What if the file doing patches from is actually visible on a web page? Or better yet, if the patch command itself was modified such that at the same time it applies a patch, the source and the results were added to a MySQL server which in turn shows as a web page? His patch file already has a bunch of patches glorped together. In theory they could be separated again by parsing the mail headers. I suppose that would be less work for Linus... The point is, the difference between the patches WE get and the patches LINUS gets is the granularity. He's constantly extorting other people to watch the granularity of what they send him (small patches, each doing one thing, with good documentation as to what they do and why, in seperate messages), but what WE get is a great big diff about once a week. So what I'm trying to figure out is if we can impose on Linus to cc: a mailing list on the stream of individual patch messages he's applying to his tree, so we can follow it better. And he IS willing to do a LITTLE work for us. He's issuing changelogs now. This would be significantly less effort than that... MySQL is overkill, and since these things started as mail messages why should they be converted into a foriegn format? There's threaded archivers for mailing lists and newsgroups and stuff already, why reinvent the wheel? Rob - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: Why use threads ( was: Alan Cox quote?)
On Wed, Jun 20, 2001 at 03:18:58PM -0700, David Schwartz wrote: As I said, you don't want to use one thread for each client. You use, say, 10 threads for the 16,000 clients. That way, if an occasional client ambushes a thread (say by reading a file off an NFS server or by using some infrequently used code that was swapped to a busy disk), your server will keep on humming. This same approach can easily be used by multiple processes. Theoretically, pretty much everything you can do with a thread pool architecture can be done with a process pool architecture. Sharing file descriptors can be much harder. Some thread architectures (notably pthreads) go out of their way to make some things hard for thread pools (like try implementing Samba, since all the threads share there uids). But this is more about bad threading specifications and implementations than threading itself. I don't see what is gained by using threads over processes for such an architecture. I'll flip this back at you and ask you what's gained by using processes over threads. It's certainly more difficult to implement a process pool architecture than a thread pool architecture. Theoretically, performance for a thread pool architecture will be slightly better (on some architectures at least) due to the identical vm views. With a process pool architecture, you might have more control over what you share. But this doesn't actually gain you anything because as long as you share more than a small amount, you still have to consider your entire environment tainted if one execution vehicle goes out of control. I would be very interested in seeing, for example, a web server based on a 'process pool' design. Does anybody know of any? DS - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Alan == Alan Cox [EMAIL PROTECTED] writes: But that foregoes the point that the code is far more complex and harder to make 'obviously correct', a concept that *does* translate well to userspace. Alan There I disagree. Threads introduce parallelism that the Alan majority of user space programmers have trouble getting right Alan (not that C is helpful here). Alan A threaded program has a set of extremely complex hard to repeat Alan timing based behaviour dependancies. An unthreaded app almost Alan always does the same thing on the same input. From a Alan verification and coverage point of view that is incredibly Alan important. Not to mention how complex it is to get locking right in an efficient manner. Programming threads is not that much different from kernel SMP programming, except that in userland you get a core dump and retry, in the kernel you get an OOPS and an fsck and retry. For some reason CS professors decided that threads were the cool thing to do and started teaching all their students to program threadded applications. Unfortunately they did forget to teach about the caveats and all the funny side effects that generally mostly threadded apps a really stupid idea. It's been like this for some years now, one would expect a switch to another buzzword of the year paradigm from them soon. Jes - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Wednesday 20 June 2001 11:33, Alexander Viro wrote: On 20 Jun 2001, Jes Sorensen wrote: Not to mention how complex it is to get locking right in an efficient manner. Programming threads is not that much different from kernel SMP programming, except that in userland you get a core dump and retry, in the kernel you get an OOPS and an fsck and retry. Arrgh. As long as we have that SMP makes locking harder myth floating around we _will_ get problems. Kernel UP programming is not different from SMP one. It is multithreaded. And amount of genuine SMP bugs is very small compared to ones that had been there on UP since way back. And yes, programming threads is the same thing. No arguments here. Hopefully in 2.5 we'll get the pre-emptive UP patch in that enables the SMP locks on UP and finally clean it all out by exposing the bugs to the main user base. As for multi-threaded programming being hard, people are unfamiliar with it. Any programming is hard the first time. And almost anybody's first attempt at it is going to suck. (Dig out the linux kernel 0.02 sources sometime and compare them with what we have today.) The more experience you get with it, the better you are. Encouraging people to stay away from it rather than learn to do it RIGHT is the wrong attitude. People will figure out that using 1000 threads when you need 3 isn't the best way to go, that locking is an expense but failing to lock is worse, how to spot race conditions (the same old security mindset except you don't need a malicious third party to get bitten by it.) And they'll learn it the way I did, and the way everybody does. Do it wrong repeatedly. Make every mistake there is, hard. Get burned, rewrite it from scratch four times until you figure out how to design it right, spend long weekends looking for subtle little EVIL bugs you can't reproduce but which bite you the instant you stop looking for them, learn to play hot potato with volatile data you have to manipulate atomically... Everybody starts as a bad programmer. Some of us get it out of our systems when we're 12. Others decide programming is lucrative when they're 35 and inflict their hello world opus upon their coworkers. Me, I wrote a disk editor and a bunch of bbses in 5th and 6th grade back on the C64 that will never see the light of day. And yes they sucked. But I'm still proud of them. And I wrote awful multithreaded code back on OS/2, and can now write decent threading code because I've learned a large number of things to avoid doing. And I take proper care because I know how hard it is to FIND one of these if you do wind up doing it. I've done it. Once for two consecutive weeks on the same #%*(#% bug. There's no WAY somebody's going to pick that sort of thing up in a programming course, or from advice from experienced people. We never listen to advice from experienced programmers. If we did we'd still be using Fortran for everything... Rob - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Al == Alexander Viro [EMAIL PROTECTED] writes: Al On 20 Jun 2001, Jes Sorensen wrote: Not to mention how complex it is to get locking right in an efficient manner. Programming threads is not that much different from kernel SMP programming, except that in userland you get a core dump and retry, in the kernel you get an OOPS and an fsck and retry. Al Arrgh. As long as we have that SMP makes locking harder myth Al floating around we _will_ get problems. Kernel UP programming is Al not different from SMP one. It is multithreaded. And amount of Al genuine SMP bugs is very small compared to ones that had been Al there on UP since way back. And yes, programming threads is the Al same thing. No arguments here. Call it SMP or kernel threading, I don't really care, it's the same thing. My point is that in the kernel you must take threading/SMP into account when coding and yes it's not trivial to do it efficiently (though often fairly easy to do it inefficiently) and the same applies to userland threads. Userland threads are just not some chest of gold that just opens up a free path to paradise as most CS teachers seems to promote it as being. Jes - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Oh, how I would love for select() and poll() to work on files... or for any other working AIO mothods to be present. What would get broken if things were changed to let select() work for filesystem fds? It does. They are always readable. If it were to trigger prefetching in that case then I suspect no harm would occur - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tue, Jun 19, 2001 at 08:04:42PM -0700, Larry McVoy wrote: > On the other hand, the fact that it doesn't exist on other platforms sort > of means that it isn't going anywhere. In a sick sort of way, the most > likely way to make this happen is to get Microsoft to do it and then Linux > will do it as well and then the Solaris jocks will also fall in line. The > only problem with that is that Microsoft can't design an OS interface to save > their lives, so maybe Linux _should_ do it first. world domination has to start somewhere... -- John Lenton ([EMAIL PROTECTED]) -- Random fortune: Tu amigo tiene un amigo, y el amigo de tu amigo tiene otro amigo; por consiguiente, sé discreto. -- Proverbio judío. PGP signature
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tue, Jun 19, 2001 at 10:57:38PM -0400, Michael Rothwell wrote: > On 19 Jun 2001 20:01:56 +0100, Alan Cox wrote: > > > Linux inherits several unix properties which are not friendly to good state > > based programming - lack of good AIO for one. > > Oh, how I would love for select() and poll() to work on files... or for > any other working AIO mothods to be present. > > What would get broken if things were changed to let select() work for > filesystem fds? I asked Linus for this a long time ago and he pointed out that you couldn't make it work over NFS, at least not nicely. It does seem like that could be worked around by having a "poll daemon" which knew about all the things being waited on and checked them. Or something. I'd like it too. And I'd like a callback for iocompletion, a way to do preread(fd, len). On the other hand, the fact that it doesn't exist on other platforms sort of means that it isn't going anywhere. In a sick sort of way, the most likely way to make this happen is to get Microsoft to do it and then Linux will do it as well and then the Solaris jocks will also fall in line. The only problem with that is that Microsoft can't design an OS interface to save their lives, so maybe Linux _should_ do it first. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On 19 Jun 2001 20:01:56 +0100, Alan Cox wrote: > Linux inherits several unix properties which are not friendly to good state > based programming - lack of good AIO for one. Oh, how I would love for select() and poll() to work on files... or for any other working AIO mothods to be present. What would get broken if things were changed to let select() work for filesystem fds? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tue, Jun 19, 2001 at 06:30:54PM -0700, Ben Greear wrote:
> Yeah, and we are young and prolific too, so you better watch out! :)
Prolific != competent.
--
Mike Castle [EMAIL PROTECTED] www.netcom.com/~dalgoda/
We are all of us living in the shadow of Manhattan. -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
Larry McVoy wrote: > > Good question but I doubt we're going to get anywhere. Anyone who thinks > that 73MB of RAM is an OK thing to waste on window system is probably a > died-in-the-wool Java programmer and could care less about performance, > system design, or any elegance whatsoever. Bleh, don't go taking cheap shots at Java! Java is good at what it does, memory and performance be damned! > It pains me to believe that people like this exist but there they are. Yeah, and we are young and prolific too, so you better watch out! :) Ben -- Ben Greear <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tue, Jun 19, 2001 at 05:19:45PM -0700, Mike Castle wrote: > On Tue, Jun 19, 2001 at 04:56:16PM -0700, Jonathan Lundell wrote: > > But so what? That's $16 worth of DRAM (I just checked). Not so bad > > *if* threads are otherwise a great solution. I grant that one might > > have a pretty tough time making the case, but again, for the right > > application, say some app with a dedicated server, 73MB isn't the end > > of the world (though I suppose it was at the time...). > > How much would 73MB of cache cost? How much would it cost to get that much > on the CPU? Good question but I doubt we're going to get anywhere. Anyone who thinks that 73MB of RAM is an OK thing to waste on window system is probably a died-in-the-wool Java programmer and could care less about performance, system design, or any elegance whatsoever. It pains me to believe that people like this exist but there they are. What can you do? As Confucius says "If I hold up three corners of a square and the student does not hold up the fourth, I do not bother to go over the point again". -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tue, Jun 19, 2001 at 04:56:16PM -0700, Jonathan Lundell wrote:
> But so what? That's $16 worth of DRAM (I just checked). Not so bad
> *if* threads are otherwise a great solution. I grant that one might
> have a pretty tough time making the case, but again, for the right
> application, say some app with a dedicated server, 73MB isn't the end
> of the world (though I suppose it was at the time...).
How much would 73MB of cache cost? How much would it cost to get that much
on the CPU?
mrc
--
Mike Castle [EMAIL PROTECTED] www.netcom.com/~dalgoda/
We are all of us living in the shadow of Manhattan. -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Alan Cox quote? (was: Re: accounting for threads)
On Tue, 19 Jun 2001, David S. Miller wrote: > > > Don't believe me that Solaris sucks here? Run this experiment under > Solaris-latest and Linux on a sparc64 system (using lmbench): > > Under Solaris: ./lat_proc fork > Under Linux: strace -f ./lat_proc fork > > I bet the Linux case does better than the Solaris run by some orders > of magnitude. That's how poor their fork/exit/switch code is. > > It's a very impressive difference: Script started on Tue Jun 19 17:49:30 2001 [kaboom@thing2 linux]$ ./lat_proc fork Process fork+exit: 563.7778 microseconds [kaboom@thing2 linux]$ ./lat_proc fork Process fork+exit: 565.5556 microseconds [kaboom@thing2 linux]$ ./lat_proc fork Process fork+exit: 568. microseconds [kaboom@thing2 linux]$ exit Script done on Tue Jun 19 17:49:46 2001 Script started on Tue 19 Jun 2001 05:51:38 PM MDT [kaboom@thing1 solaris]$ ./lat_proc fork Process fork+exit: 4249.5000 microseconds [kaboom@thing1 solaris]$ ./lat_proc fork Process fork+exit: 4212.5000 microseconds [kaboom@thing1 solaris]$ ./lat_proc fork Process fork+exit: 4241. microseconds [kaboom@thing1 solaris]$ exit script done on Tue 19 Jun 2001 05:52:19 PM MDT thing1 and thing2 are identical Sun Blade 100s. thing1 is running Solaris 8 (04/01 release), while thing2 is running 2.4.4 (Debian/unstable). later, chris -- Chris Ricker [EMAIL PROTECTED] [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
