Re: Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

2017-06-28 Thread Andrei Hurynovich
Thank you Valdis.

Yes, I'm basically getting what I want - the RT proc never ever gives up 
to the system. There are a plenty of cores left to run non-rt tasks on 
the machine.


The question is why this old 2.6 kernel decide that it needs per-cpu 
events and kblockd tasks.

Maybe someone can give a hint in what subsystem's documentation can I 
find anything about workqueue tasks.

/Documentation/kernel-per-CPU-kthreads.txt is great, but described 
controls appeared only in 3.10.x :)


On 06/28/2017 01:04 PM, valdis.kletni...@vt.edu wrote:
> On Wed, 28 Jun 2017 08:39:07 -0500, Andrei Hurynovich said:
>> We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are
>> NEVER interrupted.
>> According to /proc/sched_debug, it seems that kernel still schedules
>> some SCHED_OTHER(e.g. non-realtime) kernel tasks to isolated cpus - for
>> example cpu 18 get tasks events/18 and kblockd/18 that are stuck in
>> runnable(but not running state), so those kernel processes never got a
>> single time slice because our realtime process hogs 100% of cpu.
> This is what happens when you have a priority inversion - when you tell the 
> system
> to give 100% to a process, you shouldn't be surprised when other tasks don't
> get any service.
>
>> The question is: Is it possible to never schedule kernel tasks on
>> selected cpus?
> Only if the userspace process on that CPU never makes system calls - which is
> very unlikely if the process has actual real-time requirements.
>
> Also, if your "real-time" process is taking 100% of the CPU, you have a 
> disaster
> waiting to happen.  You have zero headroom for dealing with unexpected events.
> Thought experiment:  What happens if your real-world part of the system has
> an unexpected error, that requires 1% of a CPU for error recovery?  You are
> forced to either ignore the error or miss a real-time deadline.
>
> You might want to think about dividing up your process into 2 parts - one that
> handles the *actual* real-time work and only uses (for example) 20-30% of a
> CPU, and the parts that don't have actual real-time constraints that can then
> run with the rest of the available CPU, but allow other threads (such as 
> kernel)
> to execute as well.
>

-- 
Thanks,
Andrei Hurynovich

Charlesworth Research LLC.
http://www.charlesworthresearch.com/


___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies


Kernel schedules kernel tasks on isolated cpus, SCHED_FIFO prevents kernel tasks from running

2017-06-28 Thread Andrei Hurynovich
Hi.

We are trying to build realtime(-ish) system based on rhel6(kernel 
2.6.32-642.1.1.el6.x86_64).

We used isolcpus to remove some cpus from process 
scheduling(isolcpus=2-19 nohz_full=2-19 rcu_nocbs=2-19).

We spin off a program thread that set's its cpu affinity to one of those 
isolated cpus, sets its scheduling chass to SCHED_FIFO, spins in a loop 
and never sched_yield()-s to the kernel.

We set sysctl kernel.sched_rt_runtime_us = -1 so realtime threads are 
NEVER interrupted.

We are observing that the program thread is indeed realtime and is never 
interrupted.

After some time working like this, the system becomes irresponsive - ssh 
connections start failing with timeout, existing connections hang when 
trying to read/write to physical disks(reading procfs or writing to 
tmpfs is unaffected).

***
According to /proc/sched_debug, it seems that kernel still schedules 
some SCHED_OTHER(e.g. non-realtime) kernel tasks to isolated cpus - for 
example cpu 18 get tasks events/18 and kblockd/18 that are stuck in 
runnable(but not running state), so those kernel processes never got a 
single time slice because our realtime process hogs 100% of cpu.

And kernel/18 and kblockd/18 never migrate to other cpus because these 
tasks are pinned to cpu 18.
***

Check please these /proc/sched_debug snapshots, you can see that 
events/18 and kblockd/18 sum-exec counters are not increasing:
https://gist.github.com/altmind/5cf4aad87a4a082441c1ca9378a06154




The question is: Is it possible to never schedule kernel tasks on 
selected cpus?

-- 
Thanks,
Andrei Hurynovich


___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies