Odd 'hang' trying to ls a directory ...
Running -STABLE from May 15th, if I do an 'ls /usr', it works ... if I do
an 'ls /vm', it hangs .. the machine doesn't hang, as far as I can tell,
all the web sites are accessible, and there are no errors in
/var/log/messages .. I have soft-updates disabled on that file system, and
iostat is showing the drive array to be quite idle:
tty da0pass0pass1 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 35 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 13 0 14 1 73
08 9.62 30 0.28 0.00 0 0.00 0.00 0 0.00 4 0 3 0 93
08 15.00 2 0.03 0.00 0 0.00 0.00 0 0.00 7 0 1 0 92
08 15.00 4 0.06 0.00 0 0.00 0.00 0 0.00 3 0 1 0 96
08 15.77 30 0.46 0.00 0 0.00 0.00 0 0.00 0 0 2 0 97
I can run 'df', but I can't run 'df -t ufs', as that too just hangs ...
According to ps axl, I have the following in WCHAN states:
neptune# ps axl | awk '{print $9}' | sort | uniq -c
6 -
1 WCHAN
207 accept
103 inode
2 kqread
412 lockf
68 nanslp
8 pause
1 piperd
13 poll
3 psleep
58 sbwait
1 sched
626 select
1 syncer
3 ttyin
1 vlruwt
8 wait
Thoughts? Does anything above look like its 'hanging' or 'locking' the file
system? A few minutes later, there are a few variations:
neptune# ps axl | awk '{print $9}' | sort | uniq -c
6 -
1 WCHAN
207 accept
129 inode
426 lockf
67 nanslp
8 pause
1 piperd
13 poll
3 psleep
61 sbwait
1 sched
622 select
1 syncer
3 ttyin
1 vlruwt
9 wait
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: system slowdown - vnode related
On Mon, 25 May 2003, Mike Harding wrote: I'm running a very recent RELENG-4 - but I had a suspicion that this was unionfs related, so I unmounted the /usr/ports union mounts under a jail in case this was causing the problem, and haven't seen the problem since. It's possible I accidently reverted to 4.8 when I built a release, but I don't see how... 'K, I wouldn't touch anything less then 4.8-STABLE ... the last set of vnode related patches that I'm aware of were made *post* 4.8-RELEASE, which, I believe, won't be included in RELENG-4 ... ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: system slowdown - vnode related
On Mon, 26 May 2003, Mike Harding wrote:
Er - are any changes made to RELENG_4_8 that aren't made to RELENG_4? I
thought it was the other way around - that 4_8 only got _some_ of the
changes to RELENG_4...
Ack, my fault ... sorry, wasn't thinking :( RELENG_4 is correct ... I
should have confirmed my settings before blathering on ...
One of the scripts I used extensively while debugging this ... a quite
simple one .. was:
#!/bin/tcsh
while ( 1 )
echo `sysctl debug.numvnodes` - `sysctl debug.freevnodes` - `sysctl
debug.vnlru_nowhere` - `ps auxl | grep vnlru | grep -v grep | awk '{print $20}'`
sleep 10
end
which outputs this:
debug.numvnodes: 463421 - debug.freevnodes: 220349 - debug.vnlru_nowhere: 3 - vlruwt
I have my maxvnodes set to 512k right now ... now, when the server hung,
the output would look something like (this would be with 'default' vnodes):
debug.numvnodes: 199252 - debug.freevnodes: 23 - debug.vnlru_nowhere: 12 - vlrup
with the critical bit being the vlruwt - vlrup change ...
with unionfs, you are using two vnodes per file, instead of one in
non-union mode, which is why I went to 512k vs the default of ~256k vnodes
... it doesn't *fix* the problem, it only reduces its occurance ...
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: system slowdown - vnode related
Ack, only 512Meg of memory? :( k, now you are beyond me on this one ...
I'm running 4Gig on our server, with 2Gig allocated to kernel memory ...
only thing I can suggest is try slowly incrementing up your maxvnodes and
see if that helps, but, I'm not sure where your upper threshhold is with
512Meg of RAM ...
On Mon, 26 May 2003, Mike Harding wrote:
On my sytem, with 512 meg of memory, I have the following (default)
vnode related values:
bash-2.05b$ sysctl -a | grep vnode
kern.maxvnodes: 36079
kern.minvnodes: 9019
vm.stats.vm.v_vnodein: 140817
vm.stats.vm.v_vnodeout: 0
vm.stats.vm.v_vnodepgsin: 543264
vm.stats.vm.v_vnodepgsout: 0
debug.sizeof.vnode: 168
debug.numvnodes: 33711
debug.wantfreevnodes: 25
debug.freevnodes: 5823
...is this really low? Is this something that should go into
tuning(7)? I searched on google and found basically nothing related to
adjust vnodes - although I am admittedly flogging the system - I have
leafnode+ running, a mirrored CVS tree, an experimental CVS tree,
mount_union'd /usr/ports in a jaile, and so on. Damn those $1 a
gigabyte drives!
On Mon, 2003-05-26 at 09:12, Marc G. Fournier wrote:
On Mon, 26 May 2003, Mike Harding wrote:
Er - are any changes made to RELENG_4_8 that aren't made to RELENG_4? I
thought it was the other way around - that 4_8 only got _some_ of the
changes to RELENG_4...
Ack, my fault ... sorry, wasn't thinking :( RELENG_4 is correct ... I
should have confirmed my settings before blathering on ...
One of the scripts I used extensively while debugging this ... a quite
simple one .. was:
#!/bin/tcsh
while ( 1 )
echo `sysctl debug.numvnodes` - `sysctl debug.freevnodes` - `sysctl
debug.vnlru_nowhere` - `ps auxl | grep vnlru | grep -v grep | awk '{print $20}'`
sleep 10
end
which outputs this:
debug.numvnodes: 463421 - debug.freevnodes: 220349 - debug.vnlru_nowhere: 3 -
vlruwt
I have my maxvnodes set to 512k right now ... now, when the server hung,
the output would look something like (this would be with 'default' vnodes):
debug.numvnodes: 199252 - debug.freevnodes: 23 - debug.vnlru_nowhere: 12 - vlrup
with the critical bit being the vlruwt - vlrup change ...
with unionfs, you are using two vnodes per file, instead of one in
non-union mode, which is why I went to 512k vs the default of ~256k vnodes
... it doesn't *fix* the problem, it only reduces its occurance ...
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: portupgrade issue 4.8-STABLE
On Tuesday 27 May 2003 06:22 am, Rob B wrote: Hi all, Having a bit of a problem with portupgrade on a -STABLE machine: [EMAIL PROTECTED]/root: portupgrade -aRn --- Session started at: Tue, 27 May 2003 22:54:00 +1000 closed stream --- Session ended at: Tue, 27 May 2003 22:54:01 +1000 (consumed 00:00:01) [EMAIL PROTECTED]/root: pkgdb -F does the same thing. Any clues? Did you rebuild the indexes that portupgrade needs after you cvsup'ed ports-all? Kent cheers, Rob [EMAIL PROTECTED]/root: uname -a FreeBSD erwin.number6 4.8-STABLE FreeBSD 4.8-STABLE #1: Wed May 21 00:52:39 EST 2003 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/ERWIN alpha -- Kent Stewart Richland, WA http://users.owt.com/kstewart/index.html ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Problem with bind8 and ports
- Original Message - From: Doug Barton [EMAIL PROTECTED] To: Hannes Adollarson [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Tuesday, May 27, 2003 12:37 AM Subject: Re: Problem with bind8 and ports On Mon, 26 May 2003, Hannes Adollarson wrote: Hi When I install bind8 through ports it doesn't create the dir named 'namedb' and the needed files that it's supposed to do according to the handbook. Has anyother experienced this? If you use the port version without flags, it will look for files in /usr/local/etc (as indicated in the man page, although you probably can't see it because the system man page for named is in the way). The port definitely doesn't create a named.conf file for you, although the one in /etc/namedb/ will work as a start. You could also create a symlink in /usr/local/etc to /etc/namedb/named.conf. If you'd prefer to replace the system version of bind with the one from the port, then use the PORT_REPLACES_BASE_BIND8 compile option for the port. Hope this helps, Doug -- Hi That option was already included in the makefile... Hannes This .signature sanitized for your protection ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: system slowdown - vnode related
I'm a little confused. What state is the vnlru kernel thread in? It
sounds like vnlru must be stuck.
Note that you can gdb the live kernel and get a stack backtrace of any
stuck process.
gdb -k /kernel.debug /dev/mem (or whatever)
proc N (e.g. vnlru's pid)
back
All the processes stuck in 'inode' are likely associated with the
problem, but if that is what is causing vnlru to be stuck I would expect
vnlru itself to be stuck in 'inode'.
unionfs is probably responsible. I would not be surprised at all if
unionfs is causing a deadlock somewhere which is creating a chain of
processes stuck in 'inode' which is in turn causing vnlru to get stuck.
-Matt
Matthew Dillon
[EMAIL PROTECTED]
:
:On Mon, 26 May 2003, Mike Harding wrote:
:
: Er - are any changes made to RELENG_4_8 that aren't made to RELENG_4? I
: thought it was the other way around - that 4_8 only got _some_ of the
: changes to RELENG_4...
:
:Ack, my fault ... sorry, wasn't thinking :( RELENG_4 is correct ... I
:should have confirmed my settings before blathering on ...
:
:One of the scripts I used extensively while debugging this ... a quite
:simple one .. was:
:
:#!/bin/tcsh
:while ( 1 )
: echo `sysctl debug.numvnodes` - `sysctl debug.freevnodes` - `sysctl
debug.vnlru_nowhere` - `ps auxl | grep vnlru | grep -v grep | awk '{print $20}'`
: sleep 10
:end
:
:which outputs this:
:
:debug.numvnodes: 463421 - debug.freevnodes: 220349 - debug.vnlru_nowhere: 3 - vlruwt
:
:I have my maxvnodes set to 512k right now ... now, when the server hung,
:the output would look something like (this would be with 'default' vnodes):
:
:debug.numvnodes: 199252 - debug.freevnodes: 23 - debug.vnlru_nowhere: 12 - vlrup
:
:with the critical bit being the vlruwt - vlrup change ...
:
:with unionfs, you are using two vnodes per file, instead of one in
:non-union mode, which is why I went to 512k vs the default of ~256k vnodes
:... it doesn't *fix* the problem, it only reduces its occurance ...
:___
:[EMAIL PROTECTED] mailing list
:http://lists.freebsd.org/mailman/listinfo/freebsd-stable
:To unsubscribe, send any mail to [EMAIL PROTECTED]
:
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: system slowdown - vnode related
I'll try this if I can tickle the bug again.
I may have just run out of freevnodes - I only have about 1-2000 free
right now. I was just surprised because I have never seen a reference
to tuning this sysctl.
- Mike H.
On Tue, 2003-05-27 at 11:09, Matthew Dillon wrote:
I'm a little confused. What state is the vnlru kernel thread in? It
sounds like vnlru must be stuck.
Note that you can gdb the live kernel and get a stack backtrace of any
stuck process.
gdb -k /kernel.debug /dev/mem (or whatever)
proc N(e.g. vnlru's pid)
back
All the processes stuck in 'inode' are likely associated with the
problem, but if that is what is causing vnlru to be stuck I would expect
vnlru itself to be stuck in 'inode'.
unionfs is probably responsible. I would not be surprised at all if
unionfs is causing a deadlock somewhere which is creating a chain of
processes stuck in 'inode' which is in turn causing vnlru to get stuck.
-Matt
Matthew Dillon
[EMAIL PROTECTED]
:
:On Mon, 26 May 2003, Mike Harding wrote:
:
: Er - are any changes made to RELENG_4_8 that aren't made to RELENG_4? I
: thought it was the other way around - that 4_8 only got _some_ of the
: changes to RELENG_4...
:
:Ack, my fault ... sorry, wasn't thinking :( RELENG_4 is correct ... I
:should have confirmed my settings before blathering on ...
:
:One of the scripts I used extensively while debugging this ... a quite
:simple one .. was:
:
:#!/bin/tcsh
:while ( 1 )
: echo `sysctl debug.numvnodes` - `sysctl debug.freevnodes` - `sysctl
debug.vnlru_nowhere` - `ps auxl | grep vnlru | grep -v grep | awk '{print $20}'`
: sleep 10
:end
:
:which outputs this:
:
:debug.numvnodes: 463421 - debug.freevnodes: 220349 - debug.vnlru_nowhere: 3 - vlruwt
:
:I have my maxvnodes set to 512k right now ... now, when the server hung,
:the output would look something like (this would be with 'default' vnodes):
:
:debug.numvnodes: 199252 - debug.freevnodes: 23 - debug.vnlru_nowhere: 12 - vlrup
:
:with the critical bit being the vlruwt - vlrup change ...
:
:with unionfs, you are using two vnodes per file, instead of one in
:non-union mode, which is why I went to 512k vs the default of ~256k vnodes
:... it doesn't *fix* the problem, it only reduces its occurance ...
:___
:[EMAIL PROTECTED] mailing list
:http://lists.freebsd.org/mailman/listinfo/freebsd-stable
:To unsubscribe, send any mail to [EMAIL PROTECTED]
:
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UDF filesystem on 4-STABLE?
On Sunday 25 May 2003 17:20, Warren Toomey wrote: Hi all, I'm trying to find out if UDF support can be added to 4-STABLE. The web page http://people.freebsd.org/~scottl/udf/ says that such a thing can be found at http://www.softweyr.com/asmodai/projects/udf, but the latter URL is broken. Any pointers will be greatly appreciated. I'll try to get it back on-line in the next day or so. It's on a disk drive on a shelf in my office at home. I'll get the bits pushed onto the new, improved softweyr.com server as soon as I track down the disk drive. Moving is hell, you know. -- Where am I, and what am I doing in this handbasket? Wes Peters [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: USB patch. Scanner Epson.
On Tuesday 27 May 2003 03:05, Cristiano Deana wrote: Hi, i wrote this very simple patch to let USB Epson Scanner 1250 te be recognize by freebsd. Tested on -stable and -current Is there some [EMAIL PROTECTED] who i should send the patch to MFC? Sure, send it here and I'll take a look at it. I've been meaning to buy a scanner anyhow... -- Where am I, and what am I doing in this handbasket? Wes Peters [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: system slowdown - vnode related
:I'll try this if I can tickle the bug again. : :I may have just run out of freevnodes - I only have about 1-2000 free :right now. I was just surprised because I have never seen a reference :to tuning this sysctl. : :- Mike H. The vnode subsystem is *VERY* sensitive to running out of KVM, meaning that setting too high a kern.maxvnodes value is virtually guarenteed to lockup the system under certain circumstances. If you can reliably reproduce the lockup with maxvnodes set fairly low (e.g. less then 100,000) then it ought to be easier to track the deadlock down. Historically speaking systems did not have enough physical memory to actually run out of vnodes.. they would run out of physical memory first which would cause VM pages to be reused and their underlying vnodes deallocated when the last page went away. Hence the amount of KVM being used to manage vnodes (vnode and inode structures) was kept under control. But today's Intel systems have far more physical memory relative to available KVM and it is possible for the vnode management to run out of KVM before the VM system runs out of physical memory. The vnlru kernel thread is an attempt to control this problem but it has had only mixed success in complex vnode management situations like unionfs where an operation on a vnode may cause accesses to additional underlying vnodes. In otherwords, vnlru can potentially shoot itself in the foot in such situations while trying to flush out vnodes. -Matt ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
