Re: More IO identification problems
on 26/11/2010 21:14 Adam Vande More said the following: > Thanks, avg and jhell. hald was indeed the culprit, and a > "hal-disable-polling > --device /dev/cd0" has restored my sanity. I never liked the cd notifications > anyway. Although I do plan to get something like that into the kernel one day :-) Not soon though. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: More IO identification problems
On Fri, Nov 26, 2010 at 8:33 AM, jhell wrote: > > Perhaps it's some external component? > > E.g. hald is known to perform some disk/media checks every two seconds. > > > > As well syslod will also cause sync to happen prematurely when something > goes to log. > > syslog.conf(5): > *** > To ensure that kernel messages are written to disk promptly, > syslog.conf calls fsync(2) after writing messages from the kernel. > Other messages are not synced explicitly. You may prefix a pathname > with the minus sign, ``-'', to forego syncing the specified file > after every kernel message. Note that you might lose information if > the system crashes immediately following a write attempt. Neverthe- > less, using the ``-'' option may improve performance, especially if > the kernel is logging many messages. > *** > > This is obviously the old way of ensuring logs are written to disk > before crash but now ZFS is handled differently. Check to see if adding > '-' to your syslog entries relieves that problem as ZFS will do the > right thing to ensure your data is written to disk. > Thanks, avg and jhell. hald was indeed the culprit, and a "hal-disable-polling --device /dev/cd0" has restored my sanity. I never liked the cd notifications anyway. -- Adam Vande More ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: More IO identification problems
On 11/26/2010 08:41, Andriy Gapon wrote:
> on 26/11/2010 08:05 Adam Vande More said the following:
>> As a follow-up, after applying both patches presented in this thread(FreeBSD
>> 8.2-PRERELEASE #5: Thu Nov 25 19:14:00 CST 2010),
>>
>> http://lists.freebsd.org/pipermail/freebsd-stable/2010-November/060298.html
>>
>> "top -m io" now displays much more info and is generally consistent with
>> gstat in that IO spikes occer around the same time. However, gstat and "top
>> -m io" are still not displaying any stats for 90%+ of my hard drive
>> indicator light blinks. Since the issue effects gstat as well, it doesn't
>> seem like it could be related to ZFS. When the system is basically idling,
>> the only consistent IO related entries are these;
>>
>> 6 root 15 0 1 12 0 13 20.00%
>> {l2arc_feed_threa}
>> 6 root 15 0 1 12 0 13 20.00%
>> {arc_reclaim_thre}
>> 6 root 15 0 1 12 0 13 20.00% {zvol
>> zoot/usr/ho}
>> 6 root 15 0 1 12 0 13 20.00%
>> {txg_thread_enter}
>> 6 root 15 0 1 12 0 13 20.00%
>> {txg_thread_enter}
>>
>> Entries like these occur like clockwork on 30 sec intervals. My theory here
>> is that since the ZVOL has UFS + SU, this is causing a sync?
>>
>> What I'm trying to diagnose is a much more frequent hard drive access which
>> occurs on approximately 2 sec intervals. I timed this by pinging localhost
>> and comparing the response to the blinks. It's very consistent although not
>> completely so as once in awhile the blink occurs every second. If I listen
>> carefully I can hear increased drive activity during many of these intervals
>> so the indicator light seems to be working correctly.
>>
>> If anyone has ideas or tests they'd like me to run, it would be appreciated.
>>
>
> Perhaps it's some external component?
> E.g. hald is known to perform some disk/media checks every two seconds.
>
As well syslod will also cause sync to happen prematurely when something
goes to log.
syslog.conf(5):
***
To ensure that kernel messages are written to disk promptly,
syslog.conf calls fsync(2) after writing messages from the kernel.
Other messages are not synced explicitly. You may prefix a pathname
with the minus sign, ``-'', to forego syncing the specified file
after every kernel message. Note that you might lose information if
the system crashes immediately following a write attempt. Neverthe-
less, using the ``-'' option may improve performance, especially if
the kernel is logging many messages.
***
This is obviously the old way of ensuring logs are written to disk
before crash but now ZFS is handled differently. Check to see if adding
'-' to your syslog entries relieves that problem as ZFS will do the
right thing to ensure your data is written to disk.
--
jhell,v
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: More IO identification problems
on 26/11/2010 08:05 Adam Vande More said the following:
> As a follow-up, after applying both patches presented in this thread(FreeBSD
> 8.2-PRERELEASE #5: Thu Nov 25 19:14:00 CST 2010),
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-November/060298.html
>
> "top -m io" now displays much more info and is generally consistent with
> gstat in that IO spikes occer around the same time. However, gstat and "top
> -m io" are still not displaying any stats for 90%+ of my hard drive
> indicator light blinks. Since the issue effects gstat as well, it doesn't
> seem like it could be related to ZFS. When the system is basically idling,
> the only consistent IO related entries are these;
>
> 6 root 15 0 1 12 0 13 20.00%
> {l2arc_feed_threa}
> 6 root 15 0 1 12 0 13 20.00%
> {arc_reclaim_thre}
> 6 root 15 0 1 12 0 13 20.00% {zvol
> zoot/usr/ho}
> 6 root 15 0 1 12 0 13 20.00%
> {txg_thread_enter}
> 6 root 15 0 1 12 0 13 20.00%
> {txg_thread_enter}
>
> Entries like these occur like clockwork on 30 sec intervals. My theory here
> is that since the ZVOL has UFS + SU, this is causing a sync?
>
> What I'm trying to diagnose is a much more frequent hard drive access which
> occurs on approximately 2 sec intervals. I timed this by pinging localhost
> and comparing the response to the blinks. It's very consistent although not
> completely so as once in awhile the blink occurs every second. If I listen
> carefully I can hear increased drive activity during many of these intervals
> so the indicator light seems to be working correctly.
>
> If anyone has ideas or tests they'd like me to run, it would be appreciated.
>
Perhaps it's some external component?
E.g. hald is known to perform some disk/media checks every two seconds.
--
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
More IO identification problems
As a follow-up, after applying both patches presented in this thread(FreeBSD
8.2-PRERELEASE #5: Thu Nov 25 19:14:00 CST 2010),
http://lists.freebsd.org/pipermail/freebsd-stable/2010-November/060298.html
"top -m io" now displays much more info and is generally consistent with
gstat in that IO spikes occer around the same time. However, gstat and "top
-m io" are still not displaying any stats for 90%+ of my hard drive
indicator light blinks. Since the issue effects gstat as well, it doesn't
seem like it could be related to ZFS. When the system is basically idling,
the only consistent IO related entries are these;
6 root 15 0 1 12 0 13 20.00%
{l2arc_feed_threa}
6 root 15 0 1 12 0 13 20.00%
{arc_reclaim_thre}
6 root 15 0 1 12 0 13 20.00% {zvol
zoot/usr/ho}
6 root 15 0 1 12 0 13 20.00%
{txg_thread_enter}
6 root 15 0 1 12 0 13 20.00%
{txg_thread_enter}
Entries like these occur like clockwork on 30 sec intervals. My theory here
is that since the ZVOL has UFS + SU, this is causing a sync?
What I'm trying to diagnose is a much more frequent hard drive access which
occurs on approximately 2 sec intervals. I timed this by pinging localhost
and comparing the response to the blinks. It's very consistent although not
completely so as once in awhile the blink occurs every second. If I listen
carefully I can hear increased drive activity during many of these intervals
so the indicator light seems to be working correctly.
If anyone has ideas or tests they'd like me to run, it would be appreciated.
--
Adam Vande More
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
