Scheduler question in __wake_up_common() - Real Time Apps
I have a question about the implementation in __wake_up_common() that I'm hoping someone might know the background on. This function wakes up a specified number of tasks for a wait_queue. I'm wondering why it doesn't wake up the tasks in priority order, so that for things following wake-one semantics high priority tasks get woken up before lower priority tasks. The only thing I can think of off the top of my head is that it simplifies the O(1) implementation, but I'm wondering if maybe there is something else. Thanks, Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Xterm Hangs - Possible scheduler defect?
> What's so special about a 64-way box? They're expensive and customers don't expect a single userspace thread to tie up the other 63 CPUs no matter how buggy it is. It is intuitively obvious that a buggy kernel can bring a system to its knees, but it is not intuitively obvious that a buggy userspace app can do the same thing. It is more of a supportability issue than anything, because you expect the other processors to function properly so you can get in and live-debug the application when it hits a bug that makes it CPU-bound. This is especially important if the box is, say, in a remote jungle of China or something where you don't have access to the console. The horse is dead, so lets not beat it anymore for the time being. It is quite clear that people don't want Linux to (by default) not have the gun cocked and pointed at the application developer's feet. People who want a kernel that doesn't hang in the face of bad-acting userspace apps can change the priority of important kernel threads, which seems like a reasonable workaround for now. Regards, Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Xterm Hangs - Possible scheduler defect?
> In many Unices, crucial kernel threads run at realtime priority with a > static priority higher than is accessible to user code. Yep. > That being said, however, you've got to be a privileged user to set > real time very high priority on a thread, and if you do, you'd better > know what you're doing. Any SCHED_FIFO thread should run for a time, > then sleep for a time, or it *will* DOS everything else on the > processor. This is only true if you're not doing what you said in your first paragraph, i.e. running crucial kernel threads higher than any user thread. Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Xterm Hangs - Possible scheduler defect?
> >I'm all for allowing people to shoot themselves in the foot. That doesn't > >mean that it is OK for a single userspace thread to mess up a 64-way box. > > If root has explicitly stated that the thread in question is the highest > priority thing on the entire machine, why would it not be okay. The > fact that root made a mistake is the issue here, not that the system > didn't protect itself. Yeah, I realized when I left for lunch that this statement wasn't as clear as I would like it to be. I think what we have are the need for two levels of applications: 1. That which wishes to be the highest priority userspace application, and wishes to preempt all other userspace applications. Such an application is OK being preempted by the kernel when the kernel needs to do work. IMHO, this should be the default behavior for any SCHED_FIFO application. If one of these has a bug and goes CPU-bound, the worst it can do is prevent other apps from ever using the CPU it is on. 2. Applications which actually want to be the highest priority thing on the system, including being higher than the kernel. These applications are OK with the fact that they may cause system hangs and deadlocks, and are careful not to shoot themselves in the foot. > There are professionals who use linux for audio as well, it's not just > home systems. That said, you unify them with reasonable defaults, and > the ability for root to override them. OK. Would you say it would be a reasonable default to have SCHED_FIFO userspace threads running at a lower priority than essential kernel threads (say, the load balancer and the events thread), and give root the ability to explicitly have userspace threads preempt the kernel? Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Xterm Hangs - Possible scheduler defect?
> If you keep a learning attitude, there is a chance for this discussion > to go on. However, if you keep the "Come now, don't bullshit me, this is > a broken architecture and you're just trying to cover up" attitude, > you're just going to get discarded as a troll. I'm not trying to troll here; I suppose I'm just coming from a different background. I'll try to adjust my tone. > I personally like the linux way: "root has the ability to shoot himself > in the foot if he wants to". This is my computer, damn it, I am the one > who tells it what to do. I'm all for allowing people to shoot themselves in the foot. That doesn't mean that it is OK for a single userspace thread to mess up a 64-way box. > This is much, much better than the "users are stupid, we must protect > them from themselves" kind of way that other OS'es use. Isn't this what the kernel attempts to do in many other places? What else could possibly be the point of sending SIGSEGV and causing applications to dump core when they make a mistake referencing memory? Isn't it the kernel's job to protect one application from another? I see what you're saying about the swap daemon. How about if I make my statement less black and white. Some kernel threads should always have priority over userspace. I believe the mindset required for a home system that is doing audio recordings is different than the one for enterprise-level systems. How do we unify the two? Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Xterm Hangs - Possible scheduler defect?
> Low-latency userspace apps. The audio guys, for instance, are trying to > get latencies down to the 100us range. > > If random kernel threads can preempt userspace at any time, they wreak > havoc with latency as seen by userspace. Come now. There is no such thing as a random kernel thread. Any General Purpose kernel needs the ability to do work that keeps the entire system from grinding to a halt. Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Xterm Hangs - Possible scheduler defect?
> > Hmmm... Are you suggesting it is OK for a kernel to get nearly completely > > hosed and for not fully utilize all the processors in the system because > > of one SCHED_FIFO thread? > > Sure. You specifically directed the scheduler to run your thread at a > higher priority than anything else. The way I see it, you used root's > perogative to shoot himself in the foot. You could also have used root's > perogative to don steel toed shoes(set important kernel threads to a higher > priority) before pulling the trigger. No, I specifically directed the scheduler to run my thread at a higher priority than any other userspace application. The fact that I wrote it in userspace and not in kernel space implies that I am OK with the kernel stopping me sometimes when _it_ has work to do. If I wanted something higher priority than the kernel I would have written something in kernel space instead. > SCHED_FIFO thread are supposed to preempt > > all other userspace threads... not the kernel itself. > > Not so. The scheduler makes do distinction between user and kernel threads > of execution. That is S broken it isn't even funny. > If you think that's broken, you'll _love_ Ingo's IRQ threads. While testing > one of his recent (slightly buggy)unpriveleged-user-does-RT patches in an > IRQ threadified kernel, I ran a user SCHED_FIFO task at higher than the IRQ0 > thread... if my box had been an embeded washing machine controller instead > of a desktop box, it'd have been a classic case of "No tickie no washie" :)) Yeah, thats broken too. Perhaps I don't understand this philosophy you have where the kernel isn't more important than everything else. It seems to me like there needs to be a rigid hierarchy for scheduling, lest you get into deadlock problems: 1. Kernel preempts all. There may be some hierarchy of kernel priorities too, but it isn't important here. 2. SCHED_FIFO processes preempt all userspace applications. 3. SCHED_RR. 4. SCHED_OTHER. Under no circumstances should any single CPU-bound userspace thread completely hose a 64-way SMP box. Can somebody educate me on why it is correct to do it any other way? Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Xterm Hangs - Possible scheduler defect?
> Why would anyone write a thread than uses exactly 100% of one cpu? > It seems wrong to me. It is unsafe if they really need that much > processing power, what if an interrupt happens? Then they get slightly less. > If they got a 10% faster cpu, would this thread suddenly drop to 90% > usage (and no problems with kernel threads)? > If it stays at 100% then that suggests they are using some > sort of busy waiting which is bad programming. Get a better hw > device that will provide an interrupt at the right time, and write a > driver for > that. Or find some spot in the code where a small delay in acceptable, > and set a short timer and sleep on it. It doesn't take much to get this > kernel thread going. I would make the following assertion for any kernel: No single userspace thread of execution running on an SMP system should be able to hose a box by going CPU-bound, bug in the software or no bug. Any kernel should be able to handle this case and shift general work over to other processors. While I can't speak for all commercial Unixes, I know that HP-UX handles this case just fine. I'd be extremely surprised if Solaris and AIX didn't handle it fine too. What I can't understand is why you want to cop-out and say "Oh well this is just a bug in the application... the programmer shouldn't shoot himself in the foot." If that were the attitude that kernel programmers had, why have the kernel send SIGSEGV when applications reference invalid memory? Why not just let them corrupt the memory of other apps and possibly bring the whole system down? It is the kernel's job to protect itself and userspace applications from runaway applications whenever possible. While this might not be possible for this case on a UP box, it certainly is for an SMP box. Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Xterm Hangs - Possible scheduler defect?
> But the other side of the coin is that a SCHED_FIFO userspace task > presumably has extreme latency requirements, so it doesn't *want* to be > preempted by some routine kernel operation. People would get irritated if > we were to do that. Just to follow up a bit. People writing apps that run at SCHED_FIFO know that they aren't getting hard real-time, and they are OK with that. If they wanted something more they'd run on RTLinux. Why would it be wrong to preempt the SCHED_FIFO process in the case, assuming that it is too hard to fix a broken design that doesn't allow the necessary kernel threads to run on any CPU? Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Xterm Hangs - Possible scheduler defect?
> `xterm' is waiting for the other CPU to schedule a kernel thread (which is > bound to that CPU). Once that kernel thread has done a little bit of work, > `xterm' can terminate. > > But kernel threads don't run with realtime policy, so your userspace app > has permanently starved that kernel thread. > > It's potentially quite a problem, really. For example it could prevent > various tty operations from completing, it will prevent kjournald from ever > writing back anything (on uniprocessor, etc). I've been waiting for > someone to complain ;) > > But the other side of the coin is that a SCHED_FIFO userspace task > presumably has extreme latency requirements, so it doesn't *want* to be > preempted by some routine kernel operation. People would get irritated if > we were to do that. > > So what to do? It shouldn't need to preempt the kernel operation. Why is the design such that the necessary kernel thread can't run on the other CPU? Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Xterm Hangs - Possible scheduler defect?
Hello- We have hit a defect where an exiting xterm process will hang. This is running on a 2-cpu IA-64 box. We have a multithreaded application, where one thread is SCHED_FIFO and is running with priority 98, and the other thread is just a normal SCHED_OTHER thread. The SCHED_FIFO thread is in a CPU bound tight loop, but I wouldn't expect that to cause since there are 2 CPUs. However, it does seem to cause some problems. For example, if you ssh into the system and run an Xterm using X11 forwarding, when you type "exit" in the xterm window, the window hangs and doesn't close. Killing the CPU-bound app causes the window to exit immediately. The sysrq output shows the following: xterm D a001000bef60 0 2905 2876 (NOTLB) Call Trace: [] schedule+0xca0/0x1300 sp=e00012257d20 bsp=e00012251080 [] flush_cpu_workqueue+0x1a0/0x4a0 sp=e00012257d30 bsp=e00012251020 [] flush_workqueue+0x100/0x160 sp=e00012257d90 bsp=e00012250fe8 [] flush_scheduled_work+0x20/0x40 sp=e00012257d90 bsp=e00012250fd0 [] release_dev+0x8e0/0x1100 sp=e00012257d90 bsp=e00012250f20 [] tty_release+0x30/0x60 sp=e00012257e30 bsp=e00012250ef8 [] __fput+0x330/0x340 sp=e00012257e30 bsp=e00012250ea8 [] fput+0x40/0x60 sp=e00012257e30 bsp=e00012250e88 [] filp_close+0xd0/0x160 sp=e00012257e30 bsp=e00012250e58 [] sys_close+0x140/0x1a0 sp=e00012257e30 bsp=e00012250dd8 [] ia64_ret_from_syscall+0x0/0x20 sp=e00012257e30 bsp=e00012250dd8 So it would appear that xterm is hung in close() trying to shutdown a tty. The comment says that is calling flush_scheduled_work() to "Wait for ->hangup_work and ->flip.work handlers to terminate". Perhaps there is some locking issue that is causing these to not run and complete? I'm a bit out of my space here... does anybody have any ideas? I've tried this on both 2.6.8 and 2.6.10 with the same problem resulting. Please make sure to CC me in any responses. Regards, Chad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
