Re: Re[2]: FW: Filesystem gets a huge performance boost

[Matt Dillon]

:
:Any plan to MFC? I am interesting to see it in 4.3-RELEASE.
:
:-- 
:David Xu

It will definitely not go in until after the release.  It's still 
experimental (in our tree).

-Matt


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Re[2]: FW: Filesystem gets a huge performance boost

[Bruce A. Mah]

If memory serves me right, David Xu wrote:

[dirpref stuff]

> Any plan to MFC? I am interesting to see it in 4.3-RELEASE.

I'm pretty sure it won't be in 4.3-RELEASE.  In case you didn't realize,
RELENG_4 has been in code freeze for some weeks now, preparing for a
release next week.  "Code freeze" means our release engineer only lets
critical bugfixes (and a very few selected enhancements) into the branch
pending release.  The new dirpref code hasn't even lived in HEAD long 
enough for a MFC under "normal" circumstances.

Bruce.



 PGP signature

Re[2]: FW: Filesystem gets a huge performance boost

[David Xu]

Hello Matt,

Wednesday, April 11, 2001, 2:24:35 AM, you wrote:

:>> I'm not 100% convinced about the algorithm to avoid clusters filling 
:>> up with directory-only entries (it looks like a worst-case would fill 
:>> a cluster with 50% directories and 50% files leaving a bad layout when 
:>> the directories are populated further), but then the non-dirpref 
:>> scheme has some far worse worst-case scenarios ;-)
MD> :
MD> :Just to follow up on myself... it seems the dirpref stuff was 
MD> :committed to FreeBSD this morning :-]
MD> :
MD> :-- 
MD> :Brian <[EMAIL PROTECTED]>

MD> Yup, Kirk committed it.  I really like the changes -- in the old days
MD> disk caches were tiny and directories were not well cached on top of that.
MD> It made sense to try to keep directories close to their files.

Any plan to MFC? I am interesting to see it in 4.3-RELEASE.

-- 
David Xu



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: FW: Filesystem gets a huge performance boost

[Alex Zepeda]

> Yup, Kirk committed it.  I really like the changes -- in the old days
> disk caches were tiny and directories were not well cached on top of that.
> It made sense to try to keep directories close to their files.

So I'm all excited now at the progress that ufs/ffs are making recently.  
Yay Kirk.  But this leaves one nagging question in my mind.  What sort of
intervention does one need to do to enable this newfangled wunderfs code?

- alex

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

RE: Problems with fsck after dirpref changes

[John Baldwin]

On 10-Apr-01 Niels Chr. Bank-Pedersen wrote:
> 
> Is it me fsck'ing up, or is fsck(8) lacking behind in the
> dirpref changes?
> 
> 
>   Automatic boot in progress...
>   /dev/da0s1a: BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN
> FIRST ALTERNATE
>   
>   /dev/da0s1a: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
>   /dev/da0s1a: Automatic file system check failed . . . help!
>   Enter full pathname of shell or RETURN for /bin/sh: 
>   # fsck_ffs -b 32 /
>   Alternate super block location: 32
>   ** /dev/da0s1a
>   ** Last Mounted on 
>   ** Root file system
>   ** Phase 1 - Check Blocks and Sizes
>   ** Phase 2 - Check Pathnames
>   ** Phase 3 - Check Connectivity
>   ** Phase 4 - Check Reference Counts
>   ** Phase 5 - Check Cyl groups
>   FREE BLK COUNT(S) WRONG IN SUPERBLK
>   SALVAGE? [yn] y
>   
>   2683 files, 136083 used, 399724 free (1164 frags, 49820 blocks, 0.2%
> fragmentation)
>   
>   UPDATE STANDARD SUPERBLOCK? [yn] y

You didn't want to do this.  This is probably why you panic'd.

> http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/fsck_ffs/setup.c.diff?r1=1.8&r2
> =1.9&f=h

Yep, my fsck works again (well, it doesn't blow up at least), will commit it in
a second.

-- 

John Baldwin <[EMAIL PROTECTED]> -- http://www.FreeBSD.org/~jhb/
PGP Key: http://www.baldwin.cx/~john/pgpkey.asc
"Power Users Use the Power to Serve!"  -  http://www.FreeBSD.org/

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Problems with fsck after dirpref changes

[Niels Chr. Bank-Pedersen]

Is it me fsck'ing up, or is fsck(8) lacking behind in the
dirpref changes?


  Automatic boot in progress...
  /dev/da0s1a: BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST 
ALTERNATE
  
  /dev/da0s1a: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
  /dev/da0s1a: Automatic file system check failed . . . help!
  Enter full pathname of shell or RETURN for /bin/sh: 
  # fsck_ffs -b 32 /
  Alternate super block location: 32
  ** /dev/da0s1a
  ** Last Mounted on 
  ** Root file system
  ** Phase 1 - Check Blocks and Sizes
  ** Phase 2 - Check Pathnames
  ** Phase 3 - Check Connectivity
  ** Phase 4 - Check Reference Counts
  ** Phase 5 - Check Cyl groups
  FREE BLK COUNT(S) WRONG IN SUPERBLK
  SALVAGE? [yn] y
  
  2683 files, 136083 used, 399724 free (1164 frags, 49820 blocks, 0.2% fragmentation)
  
  UPDATE STANDARD SUPERBLOCK? [yn] y
  
  
  * FILE SYSTEM WAS MODIFIED *

Wonder if we should have seen something like this:

  
http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/fsck_ffs/setup.c.diff?r1=1.8&r2=1.9&f=h


BTW, the box then panic'ed right after going multiuser, but
I dunno if thats related (managed to get it running on kernel.old):

  Fatal trap 12: page fault while in kernel mode
  cpuid = 1; lapic.id = 
  fault virtual address   = 0x4
  fault code  = supervisor read, page not present
  instruction pointer = 0x8:0xc021ebda
  stack pointer   = 0x10:0xdfa6ec20
  frame pointer   = 0x10:0xdfa6ec30
  code segment= base 0x0, limit 0xf, type 0x1b
  = DPL 0, pres 1, def32 1, gran 1
  processor eflags= interrupt enabled, resume, IOPL = 0
  current process = 319 (named)
  kernel: type 12 trap, code=0
  
  CPU1 stopping CPUs: 0x0001... stopped.
  Stopped at  ffs_valloc+0x8e:cmpb$0,0(%edi,%eax,1)
  db> trace
  ffs_valloc(dabeb1c0,81a4,c1c54500,dfa6ec58,dfa6edb8) at ffs_valloc+0x8e
  ufs_makeinode(81a4,dabeb1c0,dfa6eea4,dfa6eeb8) at ufs_makeinode+0x61
  ufs_create(dfa6edb8,dfa6ee2c,c01bd59f,dfa6edb8,dfa6ef80) at ufs_create+0x2b
  ufs_vnoperate(dfa6edb8,dfa6ef80,0,3,a02) at ufs_vnoperate+0x15
  vn_open(dfa6ee90,dfa6ee5c,1a4,d9d40100,0) at vn_open+0x177
  open(d9d40100,dfa6ef80,8114366,80eb062,0) at open+0xd6
  syscall(2f,2f,2f,0,80eb062) at syscall+0x405
  syscall_with_err_pushed() at syscall_with_err_pushed+0x1b


/Niels Chr.

-- 
 Niels Christian Bank-Pedersen, NCB1-RIPE.

 "Hey, are any of you guys out there actually *using* RFC 2549?"

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: FW: Filesystem gets a huge performance boost

[Matt Dillon]

:Why VMIO dir works better if directories are placed close to each other? I
:think it only makes the cache data of an individual directory stay in the
:memory longer.  Is there a way to measure the effectiveness of the disk
:drive's cache?
:
:-Zhihui

I wasn't being clear enough.  There are two different things going
on.  VMIO works better because it is capable of caching many, many
more directories using the VM page cache (all of physical memory) rather
then buffer malloc space (which is limited to a few megabytes).  The
buffer cache in general will also operate better with the improved 
locality of reference for the inodes underlying the files in the
directories.

The disk drive's cache works better because it will cache things 
immediately that the VM system caches across a longer span of time.
If you are scanning a large number of small directories you can wind up
with a situation where each small directory is being held in a 
file fragment (1K on disk).  Several small directories may wind up
together on-disk, either contiguous or very close by.  The VM page 
cache cannot cache an entire filesystem block (potentially holdling
8 small directories) in a single operation oweing to the virtual,
file-oriented nature of the cache.  However, the disk drive's cache CAN,
and will.  The result is that you are suddenly able to read a huge number
of small directories with 1/10 the number of hard seeks on the disk that
you would otherwise have needed.

Prior to this change, all those small directories were not near each
other on-disk.  Not only was the VM cache not able to cache the 
surrounding blocks, but even if the disk drive did the data would wind
up being thrown away anyway because the surrounding blocks were not where
the other directories resided.

Disk drives are more limited by whatever seeking they have to do then by
the amount of data they are transfering.  Anything that reduces seeking
has a huge effect on performance, and this is what is accomplished by
the patch.

-Matt



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: FW: Filesystem gets a huge performance boost

[Brian Somers]

> Why VMIO dir works better if directories are placed close to each other? I
> think it only makes the cache data of an individual directory stay in the
> memory longer.  Is there a way to measure the effectiveness of the disk
> drive's cache?

The real performance gain is seen when doing stuff with large 
directory hierarchies such as /usr/ports or (I think) a squid cache.  
The close proximity of the directories means they can be read/written 
far more quickly than before (where they were specifically placed in 
different clusters).

> -Zhihui

-- 
Brian <[EMAIL PROTECTED]>
     
Don't _EVER_ lose your sense of humour !



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: FW: Filesystem gets a huge performance boost

[Zhihui Zhang]

On Tue, 10 Apr 2001, Matt Dillon wrote:

> :> I'm not 100% convinced about the algorithm to avoid clusters filling 
> :> up with directory-only entries (it looks like a worst-case would fill 
> :> a cluster with 50% directories and 50% files leaving a bad layout when 
> :> the directories are populated further), but then the non-dirpref 
> :> scheme has some far worse worst-case scenarios ;-)
> :
> :Just to follow up on myself... it seems the dirpref stuff was 
> :committed to FreeBSD this morning :-]
> :
> :-- 
> :Brian <[EMAIL PROTECTED]>
> 
> Yup, Kirk committed it.  I really like the changes -- in the old days
> disk caches were tiny and directories were not well cached on top of that.
> It made sense to try to keep directories close to their files.
> 
> But today the proximity of a directory to its files is not really that
> important.  It is far more important for directories to have reasonable
> proximity to each other not only to improve directory scans and lookups,
> but also to improve caching.  Especially for small directories.  
> Consider that small directories are typically contained in a single
> fragment (1K).  If directories are spread all over the disk, caching
> is non-optimal.  But if they are relatively close to each other then
> both our VM cache (if vfs.vmiodirenable is set to 1) and the hard
> drive's internal cache become extremely effective.  With the added

Why VMIO dir works better if directories are placed close to each other? I
think it only makes the cache data of an individual directory stay in the
memory longer.  Is there a way to measure the effectiveness of the disk
drive's cache?

-Zhihui

> effectiveness of the caches, seeking should wind up being
> significantly reduced even for things like 'tar'.  Large directories
> also benefit, I think.
> 
> From looking at the code, I don't think fragmentation will be an issue.
> Or, to be more specific, I don't think the fact that files may not wind
> up in the same cylinder group as their directory entry is an issue.
> Either you have a huge number of directories being accessed and need
> the locality of reference within the directory space even if it costs
> some additional seeking to access underlying files, or you don't and
> the active directories all wind up being cached, removing any additional
> seeking from the equation entirely.
> 
>   -Matt
> 
> 
> To Unsubscribe: send mail to [EMAIL PROTECTED]
> with "unsubscribe freebsd-current" in the body of the message
> 


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: FW: Filesystem gets a huge performance boost

[Matt Dillon]

:> I'm not 100% convinced about the algorithm to avoid clusters filling 
:> up with directory-only entries (it looks like a worst-case would fill 
:> a cluster with 50% directories and 50% files leaving a bad layout when 
:> the directories are populated further), but then the non-dirpref 
:> scheme has some far worse worst-case scenarios ;-)
:
:Just to follow up on myself... it seems the dirpref stuff was 
:committed to FreeBSD this morning :-]
:
:-- 
:Brian <[EMAIL PROTECTED]>

Yup, Kirk committed it.  I really like the changes -- in the old days
disk caches were tiny and directories were not well cached on top of that.
It made sense to try to keep directories close to their files.

But today the proximity of a directory to its files is not really that
important.  It is far more important for directories to have reasonable
proximity to each other not only to improve directory scans and lookups,
but also to improve caching.  Especially for small directories.  
Consider that small directories are typically contained in a single
fragment (1K).  If directories are spread all over the disk, caching
is non-optimal.  But if they are relatively close to each other then
both our VM cache (if vfs.vmiodirenable is set to 1) and the hard
drive's internal cache become extremely effective.  With the added
effectiveness of the caches, seeking should wind up being
significantly reduced even for things like 'tar'.  Large directories
also benefit, I think.

From looking at the code, I don't think fragmentation will be an issue.
Or, to be more specific, I don't think the fact that files may not wind
up in the same cylinder group as their directory entry is an issue.
Either you have a huge number of directories being accessed and need
the locality of reference within the directory space even if it costs
some additional seeking to access underlying files, or you don't and
the active directories all wind up being cached, removing any additional
seeking from the equation entirely.

-Matt


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: ** HEADS UP ** portmap daemon renamed to rpcbind

[Brooks Davis]

On Mon, Apr 09, 2001 at 08:49:59PM -0700, Rodney W. Grimes wrote:
> > Question:  Does the rpcbind program in -current have the same problem
> > or has it already been fixed by whomever you imported the code from?
> > (If it hasn't been fixed I'll be happy to fix it.  I'm hoping it has,
> > though).
> 
> Given the length of time that this problem has existed some how I doubt
> it...

I'm pretty sure it hasn't given that the Solaris NIS+ implementation
(at least as of 2.6) actually embeded both the port and the IP address
of the call back TCP port when making requests for large tables.
I'm pretty sure it did this due to the fact that they were too lazy to
select the sending IP.  The really painful thing is that the IP port
pair is encoded in ascii as dotted sextuples (123.124.125.126.32.98 =
123.124.125.126:8290).

-- Brooks

-- 
Any statement of the form "X is the one, true Y" is FALSE.
PGP fingerprint 655D 519C 26A7 82E7 2529  9BF0 5D8E 8BE9 F238 1AD4

 PGP signature

Re: Panic within sound driver

[Ollivier Robert]

According to Cameron Grant:
> fix just committed, sys/dev/sound/isa/ad1816.c rev 1.18.
> 
> you must be the only freebsd user on the planet with an ad1816. :)

Works fine BTW, thanks to you two.
-- 
Ollivier ROBERT  -=-  Eurocontrol EEC/ITM  -=-  [EMAIL PROTECTED]
FreeBSD caerdonn.eurocontrol.fr 5.0-CURRENT #46: Wed Jan  3 15:52:00 CET 2001

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Panic within sound driver

[Ollivier Robert]

According to Cameron Grant:
> fix just committed, sys/dev/sound/isa/ad1816.c rev 1.18.
> 
> you must be the only freebsd user on the planet with an ad1816. :)

That's what I was thinking :)

Thanks, I'll just reboot now to test the patch.
-- 
Ollivier ROBERT  -=-  Eurocontrol EEC/ITM  -=-  [EMAIL PROTECTED]
FreeBSD caerdonn.eurocontrol.fr 5.0-CURRENT #46: Wed Jan  3 15:52:00 CET 2001

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Panic within sound driver

[Cameron Grant]

> Stopped at _mtx_lock_sleep+0x2e2: movb  0x1d5(%edx),%al
> _mtx_lock_sleep
> snd_mtxlock
> ad1816_lock

fix just committed, sys/dev/sound/isa/ad1816.c rev 1.18.

you must be the only freebsd user on the planet with an ad1816. :)

-cg



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Panic within sound driver

[George Reid]

On Tue, 10 Apr 2001, Ollivier Robert wrote:

> Stopped at _mtx_lock_sleep+0x2e2: movb  0x1d5(%edx),%al
> _mtx_lock_sleep
> snd_mtxlock
> ad1816_lock

Following patch should fix, I'll commit this to -current later.

 - greid

Index: ad1816.c
===
RCS file: /usr/home/ncvs/src/sys/dev/sound/isa/ad1816.c,v
retrieving revision 1.17
diff -u -r1.17 ad1816.c
--- ad1816.c2001/03/24 23:10:25 1.17
+++ ad1816.c2001/04/10 13:47:55
@@ -88,13 +88,13 @@
 static void
 ad1816_lock(struct ad1816_info *ad1816)
 {
-   snd_mtxlock(ad1816);
+   snd_mtxlock(ad1816->lock);
 }
 
 static void
 ad1816_unlock(struct ad1816_info *ad1816)
 {
-   snd_mtxunlock(ad1816);
+   snd_mtxunlock(ad1816->lock);
 }
 
 static int


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Panic within sound driver

[Ollivier Robert]

I systematically get the following panic since the end of March at boot
time:

Kernel trap 12 with interrupt disabled

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x65656e48XXX een8 XXX
fault code  = supervisor read, page not present
instruction pointer = 0x8:0xc01aba3a
stack pointer   = 0x10:0xc03d66ac
frame pointer   = 0x10:0xc03d66b8
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= resume, IOPL = 0
current process = 0 (swapper)
Kernel: type 12 trap, code = 0

Stopped at _mtx_lock_sleep+0x2e2: movb  0x1d5(%edx),%al
_mtx_lock_sleep
snd_mtxlock
ad1816_lock
ad1816mix_set
mixer_set
ad1816_attach
device_probe_and_attach
isa_probe_children
configure
mi_startup

Any idea?

Source from a few hours ago.

#
# CAERDONN
#
#   $Id: //depot/caerdonn/kernel/CAERDONN#14 $

machine i386
cpu I686_CPU
ident   CAERDONN
maxusers48

makeoptions DEBUG="-g"

options INET#InterNETworking
options FFS #Berkeley Fast Filesystem
options PROCFS
options COMPAT_43   #Compatible with BSD 4.3 [KEEP THIS!]
options UCONSOLE#Allow users to grab the console

options CLK_USE_TSC_CALIBRATION

options SYSVSHM
options SYSVSEM
options SYSVMSG
options SHMMAXPGS=2048

options DEVFS
options DDB
options INVARIANTS
options INVARIANT_SUPPORT

options KTRACE

options IPSEC
options IPSEC_ESP

options SOFTUPDATES

options P1003_1B
options _KPOSIX_PRIORITY_SCHEDULING
options _KPOSIX_VERSION=199309L

device  isa
device  pci

device miibus
device fxp

device  fdc

device  ata
device  atapicd

# A single entry for any of these controllers (ncr, ahb, ahc) is sufficient
# for any number of installed devices.

device  ahc

device  scbus
device  da
device  sa
device  cd   
device  pass   #CAM passthrough driver

device  atkbdc  1
device  atkbd
device  psm
device  vga
device  sc  1

device  splash

device  random

device  npx

device  sio

device  ppc
device  ppbus
device  lpt
device  ppi

device pcm

device  loop
device  ether
device  tun 2
device  pty
device  gzip# Exec gzipped a.out's
device  bpf 4
device  snp 4

-- 
Ollivier ROBERT  -=-  Eurocontrol EEC/ITM  -=-  [EMAIL PROTECTED]
FreeBSD caerdonn.eurocontrol.fr 5.0-CURRENT #46: Wed Jan  3 15:52:00 CET 2001

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: FW: Filesystem gets a huge performance boost

[Brian Somers]

[.]
> >  The second improvement, contributed by
> > [EMAIL PROTECTED], is a new directory allocation policy (codenamed
> > "dirpref"). Coupled with soft updates, the new dirpref code offers up
> > to a 60x speed increase in gluk's tests, documented here:" 
> > 
> > 
>http://groups.google.com/groups?q=dirpref&num=100&hl=en&lr=&safe=off&rnum=2&seld=905073910&ic=1
> 
> 
> I do like the dirpref stuff, but I can't comment much on it 
> except that it looks like a good change that should be fairly easy to 
> bring into FreeBSD.
> 
> I'm not 100% convinced about the algorithm to avoid clusters filling 
> up with directory-only entries (it looks like a worst-case would fill 
> a cluster with 50% directories and 50% files leaving a bad layout when 
> the directories are populated further), but then the non-dirpref 
> scheme has some far worse worst-case scenarios ;-)

Just to follow up on myself... it seems the dirpref stuff was 
committed to FreeBSD this morning :-]

-- 
Brian <[EMAIL PROTECTED]>
     
Don't _EVER_ lose your sense of humour !



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message