Re: iostat: add "sp" column for CP_SPIN
On Wed, Sep 05 2018, Solene Rapenne wrote:
> Solene Rapenne wrote:
>> Naoki Fukaumi wrote:
>> > hi tech@,
>> >
>> > new cpu state, CP_SPIN, was added,
>> > https://marc.info/?l=openbsd-cvs=152630109526317=2
>> >
>> > but there is no column for it in the header of iostat,
>> >
>> > $ iostat
>> > tty sd0 cpu
>> > tin tout KB/t t/s MB/s us ni sy in id
>> >01 11.473 0.04 0 0 0 0 0100
>> >
>> > this patch adds "sp" for CP_SPIN.
>> >
>> > --
>> > FUKAUMI Naoki
>> >
>> > Index: usr.sbin/iostat/iostat.c
>> > ===
>> > RCS file: /cvs/src/usr.sbin/iostat/iostat.c,v
>> > retrieving revision 1.40
>> > diff -u -p -u -p -r1.40 iostat.c
>> > --- usr.sbin/iostat/iostat.c 10 Feb 2018 19:49:50 - 1.40
>> > +++ usr.sbin/iostat/iostat.c 4 Sep 2018 04:19:14 -
>> > @@ -229,7 +229,7 @@ header(void)
>> >printf(" %16.16s ", cur.dk_name[i]);
>> >
>> >if (ISSET(todo, SHOW_CPU))
>> > - printf("cpu");
>> > + printf(" cpu");
>> >printf("\n");
>> >
>> >/* Sub-Headers. */
>> > @@ -254,7 +254,7 @@ header(void)
>> >printf(" KB xfr time ");
>> >
>> >if (ISSET(todo, SHOW_CPU))
>> > - printf(" us ni sy in id");
>> > + printf(" us ni sy sp in id");
>> >printf("\n");
>> > }
>> >
>>
>> diff for iostat.8
>>
>> not sure for the text though
>> is spinning time, time spent in locks?
>>
>> The text could be "CPU time waiting for locks" maybe
>>
>> Index: iostat.8
>> ===
>> RCS file: /cvs/src/usr.sbin/iostat/iostat.8,v
>> retrieving revision 1.26
>> diff -u -p -r1.26 iostat.8
>> --- iostat.817 Jan 2013 21:39:29 - 1.26
>> +++ iostat.85 Sep 2018 08:29:36 -
>> @@ -170,6 +170,8 @@ Seconds spent in disk activity
>> % of CPU time in user mode running niced processes
>> .It \
>> % of CPU time in system mode
>> +.It \
>> +% of CPU time spent spinning
>> .It \
>> % of CPU time processing interrupts
>> .It \
>
> the right diff is this one
ok jca@
> Index: iostat.8
> ===
> RCS file: /cvs/src/usr.sbin/iostat/iostat.8,v
> retrieving revision 1.26
> diff -u -p -r1.26 iostat.8
> --- iostat.817 Jan 2013 21:39:29 - 1.26
> +++ iostat.85 Sep 2018 08:29:36 -
> @@ -170,6 +170,8 @@ Seconds spent in disk activity
> % of CPU time in user mode running niced processes
> .It \
> % of CPU time in system mode
> +.It \
> +% of CPU time spent spinning
> .It \
> % of CPU time processing interrupts
> .It \
>
--
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE
Re: iostat: add "sp" column for CP_SPIN
Hi,
Committed the diff for command. Thanks.
As for the man page, it makes sense for me.
ok yasuoka
On Wed, 05 Sep 2018 10:33:37 +0200
Solene Rapenne wrote:
> Solene Rapenne wrote:
>> Naoki Fukaumi wrote:
>> > hi tech@,
>> >
>> > new cpu state, CP_SPIN, was added,
>> > https://marc.info/?l=openbsd-cvs=152630109526317=2
>> >
>> > but there is no column for it in the header of iostat,
>> >
>> > $ iostat
>> > tty sd0 cpu
>> > tin tout KB/t t/s MB/s us ni sy in id
>> >01 11.473 0.04 0 0 0 0 0100
>> >
>> > this patch adds "sp" for CP_SPIN.
>> >
>> > --
>> > FUKAUMI Naoki
>> >
>> > Index: usr.sbin/iostat/iostat.c
>> > ===
>> > RCS file: /cvs/src/usr.sbin/iostat/iostat.c,v
>> > retrieving revision 1.40
>> > diff -u -p -u -p -r1.40 iostat.c
>> > --- usr.sbin/iostat/iostat.c 10 Feb 2018 19:49:50 - 1.40
>> > +++ usr.sbin/iostat/iostat.c 4 Sep 2018 04:19:14 -
>> > @@ -229,7 +229,7 @@ header(void)
>> >printf(" %16.16s ", cur.dk_name[i]);
>> >
>> >if (ISSET(todo, SHOW_CPU))
>> > - printf("cpu");
>> > + printf(" cpu");
>> >printf("\n");
>> >
>> >/* Sub-Headers. */
>> > @@ -254,7 +254,7 @@ header(void)
>> >printf(" KB xfr time ");
>> >
>> >if (ISSET(todo, SHOW_CPU))
>> > - printf(" us ni sy in id");
>> > + printf(" us ni sy sp in id");
>> >printf("\n");
>> > }
>> >
>>
>> diff for iostat.8
>>
>> not sure for the text though
>> is spinning time, time spent in locks?
>>
>> The text could be "CPU time waiting for locks" maybe
>>
>> Index: iostat.8
>> ===
>> RCS file: /cvs/src/usr.sbin/iostat/iostat.8,v
>> retrieving revision 1.26
>> diff -u -p -r1.26 iostat.8
>> --- iostat.817 Jan 2013 21:39:29 - 1.26
>> +++ iostat.85 Sep 2018 08:29:36 -
>> @@ -170,6 +170,8 @@ Seconds spent in disk activity
>> % of CPU time in user mode running niced processes
>> .It \
>> % of CPU time in system mode
>> +.It \
>> +% of CPU time spent spinning
>> .It \
>> % of CPU time processing interrupts
>> .It \
>
> the right diff is this one
>
> Index: iostat.8
> ===
> RCS file: /cvs/src/usr.sbin/iostat/iostat.8,v
> retrieving revision 1.26
> diff -u -p -r1.26 iostat.8
> --- iostat.817 Jan 2013 21:39:29 - 1.26
> +++ iostat.85 Sep 2018 08:29:36 -
> @@ -170,6 +170,8 @@ Seconds spent in disk activity
> % of CPU time in user mode running niced processes
> .It \
> % of CPU time in system mode
> +.It \
> +% of CPU time spent spinning
> .It \
> % of CPU time processing interrupts
> .It \
>
Re: iostat: add "sp" column for CP_SPIN
Solene Rapenne wrote:
> Naoki Fukaumi wrote:
> > hi tech@,
> >
> > new cpu state, CP_SPIN, was added,
> > https://marc.info/?l=openbsd-cvs=152630109526317=2
> >
> > but there is no column for it in the header of iostat,
> >
> > $ iostat
> > tty sd0 cpu
> > tin tout KB/t t/s MB/s us ni sy in id
> >01 11.473 0.04 0 0 0 0 0100
> >
> > this patch adds "sp" for CP_SPIN.
> >
> > --
> > FUKAUMI Naoki
> >
> > Index: usr.sbin/iostat/iostat.c
> > ===
> > RCS file: /cvs/src/usr.sbin/iostat/iostat.c,v
> > retrieving revision 1.40
> > diff -u -p -u -p -r1.40 iostat.c
> > --- usr.sbin/iostat/iostat.c10 Feb 2018 19:49:50 - 1.40
> > +++ usr.sbin/iostat/iostat.c4 Sep 2018 04:19:14 -
> > @@ -229,7 +229,7 @@ header(void)
> > printf(" %16.16s ", cur.dk_name[i]);
> >
> > if (ISSET(todo, SHOW_CPU))
> > - printf("cpu");
> > + printf(" cpu");
> > printf("\n");
> >
> > /* Sub-Headers. */
> > @@ -254,7 +254,7 @@ header(void)
> > printf(" KB xfr time ");
> >
> > if (ISSET(todo, SHOW_CPU))
> > - printf(" us ni sy in id");
> > + printf(" us ni sy sp in id");
> > printf("\n");
> > }
> >
>
> diff for iostat.8
>
> not sure for the text though
> is spinning time, time spent in locks?
>
> The text could be "CPU time waiting for locks" maybe
>
> Index: iostat.8
> ===
> RCS file: /cvs/src/usr.sbin/iostat/iostat.8,v
> retrieving revision 1.26
> diff -u -p -r1.26 iostat.8
> --- iostat.817 Jan 2013 21:39:29 - 1.26
> +++ iostat.85 Sep 2018 08:29:36 -
> @@ -170,6 +170,8 @@ Seconds spent in disk activity
> % of CPU time in user mode running niced processes
> .It \
> % of CPU time in system mode
> +.It \
> +% of CPU time spent spinning
> .It \
> % of CPU time processing interrupts
> .It \
the right diff is this one
Index: iostat.8
===
RCS file: /cvs/src/usr.sbin/iostat/iostat.8,v
retrieving revision 1.26
diff -u -p -r1.26 iostat.8
--- iostat.817 Jan 2013 21:39:29 - 1.26
+++ iostat.85 Sep 2018 08:29:36 -
@@ -170,6 +170,8 @@ Seconds spent in disk activity
% of CPU time in user mode running niced processes
.It \
% of CPU time in system mode
+.It \
+% of CPU time spent spinning
.It \
% of CPU time processing interrupts
.It \
Re: iostat: add "sp" column for CP_SPIN
Naoki Fukaumi wrote:
> hi tech@,
>
> new cpu state, CP_SPIN, was added,
> https://marc.info/?l=openbsd-cvs=152630109526317=2
>
> but there is no column for it in the header of iostat,
>
> $ iostat
> tty sd0 cpu
> tin tout KB/t t/s MB/s us ni sy in id
>01 11.473 0.04 0 0 0 0 0100
>
> this patch adds "sp" for CP_SPIN.
>
> --
> FUKAUMI Naoki
>
> Index: usr.sbin/iostat/iostat.c
> ===
> RCS file: /cvs/src/usr.sbin/iostat/iostat.c,v
> retrieving revision 1.40
> diff -u -p -u -p -r1.40 iostat.c
> --- usr.sbin/iostat/iostat.c 10 Feb 2018 19:49:50 - 1.40
> +++ usr.sbin/iostat/iostat.c 4 Sep 2018 04:19:14 -
> @@ -229,7 +229,7 @@ header(void)
> printf(" %16.16s ", cur.dk_name[i]);
>
> if (ISSET(todo, SHOW_CPU))
> - printf("cpu");
> + printf(" cpu");
> printf("\n");
>
> /* Sub-Headers. */
> @@ -254,7 +254,7 @@ header(void)
> printf(" KB xfr time ");
>
> if (ISSET(todo, SHOW_CPU))
> - printf(" us ni sy in id");
> + printf(" us ni sy sp in id");
> printf("\n");
> }
>
diff for iostat.8
not sure for the text though
is spinning time, time spent in locks?
The text could be "CPU time waiting for locks" maybe
Index: iostat.8
===
RCS file: /cvs/src/usr.sbin/iostat/iostat.8,v
retrieving revision 1.26
diff -u -p -r1.26 iostat.8
--- iostat.817 Jan 2013 21:39:29 - 1.26
+++ iostat.85 Sep 2018 08:29:36 -
@@ -170,6 +170,8 @@ Seconds spent in disk activity
% of CPU time in user mode running niced processes
.It \
% of CPU time in system mode
+.It \
+% of CPU time spent spinning
.It \
% of CPU time processing interrupts
.It \
Re: iostat: add "sp" column for CP_SPIN
On 2018/09/04 16:11, Landry Breuil wrote: > On Tue, Sep 04, 2018 at 03:27:29PM +0200, Solene Rapenne wrote: > > Stuart Henderson wrote: > > > On 2018/09/04 14:22, Solene Rapenne wrote: > > > > About munin I'm trying to get a diff accepted upstream to fix cpu > > > > plugin and > > > > talk about this with kirby@. The cpu plugin uses sysctl kern.cp_time. > > > > While > > > > it's not related, I prefer to announce it here so people don't waste > > > > time > > > > fixing it again by looking at the thread :) > > > > > > Ah nice. When I worked on the initial munin port with mk I had intended > > > to try to get upstream to take the openbsd plugins separately (rather > > > than the current "copy similar OS and patch" approach), but I got fed up > > > with rrdtool performance on OpenBSD and stopped using munin before I got > > > round to it.. (and then I started using librenms and got fed up with > > > rrdtool performance once again ;) > > > > Using rrdcached the performances are really good. But that certainly depend > > on > > the number of systems. It may not scale well with more than 50 systems, > > this is > > also highly dependent on the number of plugins on each systems (as 1 value > > = 1 > > rrd file). > > Seconded, rrdcached really helps a lot until you have waaayyy too many rrds, > and then switch to a real tsdb While other time-series DBs may perform better than rrdtool, the amount of spin (kernel lock contention) seen with this relatively simple software suggests it might be a worthwhile starting point to investigate what's going on kernel-side. >(hint: influxdb is in ports) (as is prometheus. no graphite, though ;)
Re: iostat: add "sp" column for CP_SPIN
On Tue, Sep 04, 2018 at 03:27:29PM +0200, Solene Rapenne wrote: > Stuart Henderson wrote: > > On 2018/09/04 14:22, Solene Rapenne wrote: > > > About munin I'm trying to get a diff accepted upstream to fix cpu plugin > > > and > > > talk about this with kirby@. The cpu plugin uses sysctl kern.cp_time. > > > While > > > it's not related, I prefer to announce it here so people don't waste time > > > fixing it again by looking at the thread :) > > > > Ah nice. When I worked on the initial munin port with mk I had intended > > to try to get upstream to take the openbsd plugins separately (rather > > than the current "copy similar OS and patch" approach), but I got fed up > > with rrdtool performance on OpenBSD and stopped using munin before I got > > round to it.. (and then I started using librenms and got fed up with > > rrdtool performance once again ;) > > Using rrdcached the performances are really good. But that certainly depend on > the number of systems. It may not scale well with more than 50 systems, this > is > also highly dependent on the number of plugins on each systems (as 1 value = 1 > rrd file). Seconded, rrdcached really helps a lot until you have waaayyy too many rrds, and then switch to a real tsdb (hint: influxdb is in ports)
Re: iostat: add "sp" column for CP_SPIN
On 2018/09/04 15:27, Solene Rapenne wrote: > Stuart Henderson wrote: > > On 2018/09/04 14:22, Solene Rapenne wrote: > > > About munin I'm trying to get a diff accepted upstream to fix cpu plugin > > > and > > > talk about this with kirby@. The cpu plugin uses sysctl kern.cp_time. > > > While > > > it's not related, I prefer to announce it here so people don't waste time > > > fixing it again by looking at the thread :) > > > > Ah nice. When I worked on the initial munin port with mk I had intended > > to try to get upstream to take the openbsd plugins separately (rather > > than the current "copy similar OS and patch" approach), but I got fed up > > with rrdtool performance on OpenBSD and stopped using munin before I got > > round to it.. (and then I started using librenms and got fed up with > > rrdtool performance once again ;) > > Using rrdcached the performances are really good. But that certainly depend on > the number of systems. It may not scale well with more than 50 systems, this > is > also highly dependent on the number of plugins on each systems (as 1 value = 1 > rrd file). Not sure if I tried rrdcached with munin but I am using it with librenms and it's not really good here. In 9 days uptime, I have ~32h of cpu time from rrdcached, and lots of spin while the threads are actively processing. (Yes there are lots of RRDs ;) 159 threads: 2 running, 153 idle, 4 on processor up 9 days, 14:40 CPU0 states: 44.2% user, 0.0% nice, 24.5% sys, 31.3% spin, 0.0% intr, 0.0% idle CPU1 states: 45.2% user, 0.0% nice, 25.5% sys, 29.3% spin, 0.0% intr, 0.0% idle CPU2 states: 49.4% user, 0.0% nice, 24.1% sys, 26.5% spin, 0.0% intr, 0.0% idle CPU3 states: 49.0% user, 0.0% nice, 20.1% sys, 30.9% spin, 0.0% intr, 0.0% idle Memory: Real: 1847M/5197M act/tot Free: 2722M Cache: 2836M Swap: 0K/4098M
Re: iostat: add "sp" column for CP_SPIN
Stuart Henderson wrote: > On 2018/09/04 14:22, Solene Rapenne wrote: > > About munin I'm trying to get a diff accepted upstream to fix cpu plugin and > > talk about this with kirby@. The cpu plugin uses sysctl kern.cp_time. While > > it's not related, I prefer to announce it here so people don't waste time > > fixing it again by looking at the thread :) > > Ah nice. When I worked on the initial munin port with mk I had intended > to try to get upstream to take the openbsd plugins separately (rather > than the current "copy similar OS and patch" approach), but I got fed up > with rrdtool performance on OpenBSD and stopped using munin before I got > round to it.. (and then I started using librenms and got fed up with > rrdtool performance once again ;) Using rrdcached the performances are really good. But that certainly depend on the number of systems. It may not scale well with more than 50 systems, this is also highly dependent on the number of plugins on each systems (as 1 value = 1 rrd file).
Re: iostat: add "sp" column for CP_SPIN
On 2018/09/04 14:22, Solene Rapenne wrote: > About munin I'm trying to get a diff accepted upstream to fix cpu plugin and > talk about this with kirby@. The cpu plugin uses sysctl kern.cp_time. While > it's not related, I prefer to announce it here so people don't waste time > fixing it again by looking at the thread :) Ah nice. When I worked on the initial munin port with mk I had intended to try to get upstream to take the openbsd plugins separately (rather than the current "copy similar OS and patch" approach), but I got fed up with rrdtool performance on OpenBSD and stopped using munin before I got round to it.. (and then I started using librenms and got fed up with rrdtool performance once again ;)
Re: iostat: add "sp" column for CP_SPIN
Stuart Henderson wrote:
> On 2018/09/04 12:07, Solene Rapenne wrote:
> > Naoki Fukaumi wrote:
> > > hi tech@,
> > >
> > > new cpu state, CP_SPIN, was added,
> > > https://marc.info/?l=openbsd-cvs=152630109526317=2
> > >
> > > but there is no column for it in the header of iostat,
> > >
> > > $ iostat
> > > tty sd0 cpu
> > > tin tout KB/t t/s MB/s us ni sy in id
> > >01 11.473 0.04 0 0 0 0 0100
> > >
> > > this patch adds "sp" for CP_SPIN.
> > >
> > > --
> > > FUKAUMI Naoki
> > >
> > > Index: usr.sbin/iostat/iostat.c
> > > ===
> > > RCS file: /cvs/src/usr.sbin/iostat/iostat.c,v
> > > retrieving revision 1.40
> > > diff -u -p -u -p -r1.40 iostat.c
> > > --- usr.sbin/iostat/iostat.c 10 Feb 2018 19:49:50 - 1.40
> > > +++ usr.sbin/iostat/iostat.c 4 Sep 2018 04:19:14 -
> > > @@ -229,7 +229,7 @@ header(void)
> > > printf(" %16.16s ", cur.dk_name[i]);
> > >
> > > if (ISSET(todo, SHOW_CPU))
> > > - printf("cpu");
> > > + printf(" cpu");
> > > printf("\n");
> > >
> > > /* Sub-Headers. */
> > > @@ -254,7 +254,7 @@ header(void)
> > > printf(" KB xfr time ");
> > >
> > > if (ISSET(todo, SHOW_CPU))
> > > - printf(" us ni sy in id");
> > > + printf(" us ni sy sp in id");
> > > printf("\n");
> > > }
> > >
> >
> > ok solene@ for this, but the man page should be updated too, it contains the
> > list of columns displayed.
> >
>
> Is iostat(8) something where people are likely to parse output?
>
> (I first thought of sysutils/munin, iostat parsing is not enabled
> there, but there might be others).
About munin I'm trying to get a diff accepted upstream to fix cpu plugin and
talk about this with kirby@. The cpu plugin uses sysctl kern.cp_time. While
it's not related, I prefer to announce it here so people don't waste time
fixing it again by looking at the thread :)
Re: iostat: add "sp" column for CP_SPIN
On 2018/09/04 12:07, Solene Rapenne wrote:
> Naoki Fukaumi wrote:
> > hi tech@,
> >
> > new cpu state, CP_SPIN, was added,
> > https://marc.info/?l=openbsd-cvs=152630109526317=2
> >
> > but there is no column for it in the header of iostat,
> >
> > $ iostat
> > tty sd0 cpu
> > tin tout KB/t t/s MB/s us ni sy in id
> >01 11.473 0.04 0 0 0 0 0100
> >
> > this patch adds "sp" for CP_SPIN.
> >
> > --
> > FUKAUMI Naoki
> >
> > Index: usr.sbin/iostat/iostat.c
> > ===
> > RCS file: /cvs/src/usr.sbin/iostat/iostat.c,v
> > retrieving revision 1.40
> > diff -u -p -u -p -r1.40 iostat.c
> > --- usr.sbin/iostat/iostat.c10 Feb 2018 19:49:50 - 1.40
> > +++ usr.sbin/iostat/iostat.c4 Sep 2018 04:19:14 -
> > @@ -229,7 +229,7 @@ header(void)
> > printf(" %16.16s ", cur.dk_name[i]);
> >
> > if (ISSET(todo, SHOW_CPU))
> > - printf("cpu");
> > + printf(" cpu");
> > printf("\n");
> >
> > /* Sub-Headers. */
> > @@ -254,7 +254,7 @@ header(void)
> > printf(" KB xfr time ");
> >
> > if (ISSET(todo, SHOW_CPU))
> > - printf(" us ni sy in id");
> > + printf(" us ni sy sp in id");
> > printf("\n");
> > }
> >
>
> ok solene@ for this, but the man page should be updated too, it contains the
> list of columns displayed.
>
Is iostat(8) something where people are likely to parse output?
(I first thought of sysutils/munin, iostat parsing is not enabled
there, but there might be others).
Re: iostat: add "sp" column for CP_SPIN
Naoki Fukaumi wrote:
> hi tech@,
>
> new cpu state, CP_SPIN, was added,
> https://marc.info/?l=openbsd-cvs=152630109526317=2
>
> but there is no column for it in the header of iostat,
>
> $ iostat
> tty sd0 cpu
> tin tout KB/t t/s MB/s us ni sy in id
>01 11.473 0.04 0 0 0 0 0100
>
> this patch adds "sp" for CP_SPIN.
>
> --
> FUKAUMI Naoki
>
> Index: usr.sbin/iostat/iostat.c
> ===
> RCS file: /cvs/src/usr.sbin/iostat/iostat.c,v
> retrieving revision 1.40
> diff -u -p -u -p -r1.40 iostat.c
> --- usr.sbin/iostat/iostat.c 10 Feb 2018 19:49:50 - 1.40
> +++ usr.sbin/iostat/iostat.c 4 Sep 2018 04:19:14 -
> @@ -229,7 +229,7 @@ header(void)
> printf(" %16.16s ", cur.dk_name[i]);
>
> if (ISSET(todo, SHOW_CPU))
> - printf("cpu");
> + printf(" cpu");
> printf("\n");
>
> /* Sub-Headers. */
> @@ -254,7 +254,7 @@ header(void)
> printf(" KB xfr time ");
>
> if (ISSET(todo, SHOW_CPU))
> - printf(" us ni sy in id");
> + printf(" us ni sy sp in id");
> printf("\n");
> }
>
ok solene@ for this, but the man page should be updated too, it contains the
list of columns displayed.
iostat: add "sp" column for CP_SPIN
hi tech@,
new cpu state, CP_SPIN, was added,
https://marc.info/?l=openbsd-cvs=152630109526317=2
but there is no column for it in the header of iostat,
$ iostat
tty sd0 cpu
tin tout KB/t t/s MB/s us ni sy in id
01 11.473 0.04 0 0 0 0 0100
this patch adds "sp" for CP_SPIN.
--
FUKAUMI Naoki
Index: usr.sbin/iostat/iostat.c
===
RCS file: /cvs/src/usr.sbin/iostat/iostat.c,v
retrieving revision 1.40
diff -u -p -u -p -r1.40 iostat.c
--- usr.sbin/iostat/iostat.c10 Feb 2018 19:49:50 - 1.40
+++ usr.sbin/iostat/iostat.c4 Sep 2018 04:19:14 -
@@ -229,7 +229,7 @@ header(void)
printf(" %16.16s ", cur.dk_name[i]);
if (ISSET(todo, SHOW_CPU))
- printf("cpu");
+ printf(" cpu");
printf("\n");
/* Sub-Headers. */
@@ -254,7 +254,7 @@ header(void)
printf(" KB xfr time ");
if (ISSET(todo, SHOW_CPU))
- printf(" us ni sy in id");
+ printf(" us ni sy sp in id");
printf("\n");
}
