Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> On Wed, 15 May 2013, Rick Macklem wrote:
> 
> RM>Well, getdents() basically just calls kern_getdirentries() and it
> calls
> RM>VOP_READDIR() { which is called nfs_readdir() in the NFS clients }.
> RM>nfs_readdir() calls ncl_bioread() to do the real work of finding
> the
> RM>buffer cache blocks and copying the data out of them.
> RM>One thing you might check via printf()s is whether the buffer cache
> RM>has the zero'd data in it before it copies it to userland.
> 
> I now dump the data just before the call to vn_io_fault_iomove in
> ncl_bioread(). So what I do:
> 
> 1. reboot
> 2. login
> 3. ls
> -> I see that it is moving 4 blocks 4k each to the user and they look
> fine
> 4. wait half an hour
> 5. ls
> -> now there is only one block, which contains zeros starting from
> 0x200.
> 
> Note that I don't do anything else on that machine during that time.
> 
> RM>Since you get valid data sometimes and partially zero'd out data
> others,
> RM>I haven't a clue what is going on. One other person reported a
> problem
> RM>when they used a small readdirsize, but it is hard to say they saw
> the
> RM>same thing and no one else seems to be seeing this, so I have no
> idea
> RM>what it might be.
> RM>
> RM>I remember you started seeing this after an upgrade of current. Do
> you
> RM>happen to have dates (or rNN) for the old working verion vs the
> one that broke this?
> RM>(All I can think to do is scan the commits that seem to somehow
> relate
> RM> to the buffer cache or copying to userland or ???)
> 
> It looks like I had copied the old kernel before installing the new
> one
> and it is from february 5th. There is no SVN revision in it - looks
> like
> that feature was added only recently.
> 
> harti
> 
Thanks to Hartmut's testing, a patch to fix this problem has just
been committed to head as r251079. The problem was caused by
vnode_pager_setsize() being called for directories (where the
size reported by the server can be smaller than the size of the
ufs-like directory created in the client from the RPCs XDR).

r251079 will be MFC'd to stable/9 in 1 week if things go smoothly.

You might see this problem for head kernels between r248567-r251078
and stable/9 kernels from r249078 (Apr. 4) until a week from now.

Sorry for any inconvenience and thanks go to Hartmut for his help
isolating this, rick

> RM>
> RM>rick
> RM>
> RM>> harti
> RM>>
> RM>> -Original Message-
> RM>> From: Rick Macklem [mailto:rmack...@uoguelph.ca]
> RM>> Sent: Tuesday, May 14, 2013 2:50 PM
> RM>> To: Brandt, Hartmut
> RM>> Cc: curr...@freebsd.org
> RM>> Subject: Re: files disappearing from ls on NFS
> RM>>
> RM>> Hartmut Brandt wrote:
> RM>> > On Mon, 13 May 2013, Rick Macklem wrote:
> RM>> >
> RM>> > RM>Hartmut Brandt wrote:
> RM>> > RM>> On Sun, 12 May 2013, Rick Macklem wrote:
> RM>> > RM>>
> RM>> > RM>> RM>Hartmut Brandt wrote:
> RM>> > RM>> RM>> Hi,
> RM>> > RM>> RM>>
> RM>> > RM>> RM>> I've updated one of my -current machines this week
> RM>> > (previous
> RM>> > RM>> update
> RM>> > RM>> RM>> was in
> RM>> > RM>> RM>> february). Now I see a strange effect (it seems only
> on
> RM>> > NFS
> RM>> > RM>> mounts):
> RM>> > RM>> RM>> ls or
> RM>> > RM>> RM>> even echo * will list only some files (strange enough
> the
> RM>> > first
> RM>> > RM>> files
> RM>> > RM>> RM>> from
> RM>> > RM>> RM>> the normal, alphabetically ordered list). If I change
> RM>> > something
> RM>> > RM>> in the
> RM>> > RM>> RM>> directory (delete a file or create a new one) for
> some
> RM>> > time
> RM>> > the
> RM>> > RM>> RM>> complete
> RM>> > RM>> RM>> listing will appear but after sime time (seconds to a
> RM>> > minute
> RM>> > or
> RM>> > RM>> so)
> RM>> > RM>> RM>> again
> RM>> > RM>> RM>> only part of the files is listed.
> RM>> > RM>> RM>>
> RM>> > RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> RM>> > getdirentries is
> RM>> > RM>> RM>> called
> RM>>

Re: files disappearing from ls on NFS

[John Baldwin]

On Wednesday, May 15, 2013 4:38:38 am Hartmut Brandt wrote:
> On Wed, 15 May 2013, Rick Macklem wrote:
> 
> RM>Well, getdents() basically just calls kern_getdirentries() and it calls
> RM>VOP_READDIR() { which is called nfs_readdir() in the NFS clients }. 
> RM>nfs_readdir() calls ncl_bioread() to do the real work of finding the
> RM>buffer cache blocks and copying the data out of them.
> RM>One thing you might check via printf()s is whether the buffer cache
> RM>has the zero'd data in it before it copies it to userland.
> 
> I now dump the data just before the call to vn_io_fault_iomove in 
> ncl_bioread(). So what I do:
> 
> 1. reboot
> 2. login
> 3. ls
>-> I see that it is moving 4 blocks 4k each to the user and they look 
>   fine
> 4. wait half an hour
> 5. ls
>-> now there is only one block, which contains zeros starting from 
>   0x200.
> 
> Note that I don't do anything else on that machine during that time.

To be clear, you are still seeing zero'd data in your printfs?  Also, it seems 
like it is passing the wrong size now?  Can you output the various size fields 
(b_resid, n, n_direofoffset, uio_offset, and uio_resid)?

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Hartmut Brandt]

On Wed, 15 May 2013, Rick Macklem wrote:

RM>Well, getdents() basically just calls kern_getdirentries() and it calls
RM>VOP_READDIR() { which is called nfs_readdir() in the NFS clients }. 
RM>nfs_readdir() calls ncl_bioread() to do the real work of finding the
RM>buffer cache blocks and copying the data out of them.
RM>One thing you might check via printf()s is whether the buffer cache
RM>has the zero'd data in it before it copies it to userland.

I now dump the data just before the call to vn_io_fault_iomove in 
ncl_bioread(). So what I do:

1. reboot
2. login
3. ls
   -> I see that it is moving 4 blocks 4k each to the user and they look 
  fine
4. wait half an hour
5. ls
   -> now there is only one block, which contains zeros starting from 
  0x200.

Note that I don't do anything else on that machine during that time.

RM>Since you get valid data sometimes and partially zero'd out data others,
RM>I haven't a clue what is going on. One other person reported a problem
RM>when they used a small readdirsize, but it is hard to say they saw the
RM>same thing and no one else seems to be seeing this, so I have no idea
RM>what it might be.
RM>
RM>I remember you started seeing this after an upgrade of current. Do you
RM>happen to have dates (or rNN) for the old working verion vs the one that 
broke this?
RM>(All I can think to do is scan the commits that seem to somehow relate
RM> to the buffer cache or copying to userland or ???)

It looks like I had copied the old kernel before installing the new one 
and it is from february 5th. There is no SVN revision in it - looks like 
that feature was added only recently.

harti

RM>
RM>rick
RM>
RM>> harti
RM>> 
RM>> -Original Message-
RM>> From: Rick Macklem [mailto:rmack...@uoguelph.ca]
RM>> Sent: Tuesday, May 14, 2013 2:50 PM
RM>> To: Brandt, Hartmut
RM>> Cc: curr...@freebsd.org
RM>> Subject: Re: files disappearing from ls on NFS
RM>> 
RM>> Hartmut Brandt wrote:
RM>> > On Mon, 13 May 2013, Rick Macklem wrote:
RM>> >
RM>> > RM>Hartmut Brandt wrote:
RM>> > RM>> On Sun, 12 May 2013, Rick Macklem wrote:
RM>> > RM>>
RM>> > RM>> RM>Hartmut Brandt wrote:
RM>> > RM>> RM>> Hi,
RM>> > RM>> RM>>
RM>> > RM>> RM>> I've updated one of my -current machines this week
RM>> > (previous
RM>> > RM>> update
RM>> > RM>> RM>> was in
RM>> > RM>> RM>> february). Now I see a strange effect (it seems only on
RM>> > NFS
RM>> > RM>> mounts):
RM>> > RM>> RM>> ls or
RM>> > RM>> RM>> even echo * will list only some files (strange enough the
RM>> > first
RM>> > RM>> files
RM>> > RM>> RM>> from
RM>> > RM>> RM>> the normal, alphabetically ordered list). If I change
RM>> > something
RM>> > RM>> in the
RM>> > RM>> RM>> directory (delete a file or create a new one) for some
RM>> > time
RM>> > the
RM>> > RM>> RM>> complete
RM>> > RM>> RM>> listing will appear but after sime time (seconds to a
RM>> > minute
RM>> > or
RM>> > RM>> so)
RM>> > RM>> RM>> again
RM>> > RM>> RM>> only part of the files is listed.
RM>> > RM>> RM>>
RM>> > RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
RM>> > getdirentries is
RM>> > RM>> RM>> called
RM>> > RM>> RM>> only once (returning 4096). For a full listing
RM>> > getdirentries
RM>> > is
RM>> > RM>> called
RM>> > RM>> RM>> 5
RM>> > RM>> RM>> times with the last returning 0.
RM>> > RM>> RM>>
RM>> > RM>> RM>> I can still open files that are not listed if I know their
RM>> > name,
RM>> > RM>> RM>> though.
RM>> > RM>> RM>>
RM>> > RM>> RM>> The NFS server is a Windows 2008 server with an OpenText
RM>> > NFS
RM>> > RM>> Server
RM>> > RM>> RM>> which
RM>> > RM>> RM>> works without problems to all the other FreeBSD machines.
RM>> > RM>> RM>>
RM>> > RM>> RM>> So what could that be?
RM>> > RM>> RM>>
RM>> > RM>> RM>I've attached a patch that might be worth trying. It is a
RM>> > "shot in
RM>> > RM>> the dark",
RM>> > RM>> RM>but b

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> Hi Rick,
> 
> sorry for top-posting - this is Outlook :-(
> 
> Attached is the system configuration. I use this more or less
> unchanged since years. The machine is an 8-core AMD64 with 144GByte
> memory.
> 
> The nfsstats -m output for the two file systems I'm testing with is:
> 
> knopfs01:/OP_UserUnix on /home
> nfsv3,tcp,resvport,hard,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=6126856,timeout=120,retrans=2
> knopfs01:/op_software on /software
> nfsv3,tcp,resvport,hard,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=6126856,timeout=120,retrans=2
> 
> I did the tcpdump/wireshark thing and I'm puzzled that I see no
> readdir requests. I see a lookup, followed by getattr, access and
> fsstat for the directory and that's it. Looks that even after hours
> the stuff returned by getdirents(2) comes from the cache. I assume
> that the NFS client uses getattr to check whether
> the directory has changed? If I knew what happens when calling
> getdirents() I could add some debugging printfs() here and there to
> figure out...
> 
Yes. The NFS client will check the mtime on the directory to see if it has
changed and just use whatever is in the buffer cache otherwise.

Well, getdents() basically just calls kern_getdirentries() and it calls
VOP_READDIR() { which is called nfs_readdir() in the NFS clients }. 
nfs_readdir() calls ncl_bioread() to do the real work of finding the
buffer cache blocks and copying the data out of them.
One thing you might check via printf()s is whether the buffer cache
has the zero'd data in it before it copies it to userland.

Since you get valid data sometimes and partially zero'd out data others,
I haven't a clue what is going on. One other person reported a problem
when they used a small readdirsize, but it is hard to say they saw the
same thing and no one else seems to be seeing this, so I have no idea
what it might be.

I remember you started seeing this after an upgrade of current. Do you
happen to have dates (or rNN) for the old working verion vs the one that 
broke this?
(All I can think to do is scan the commits that seem to somehow relate
 to the buffer cache or copying to userland or ???)

rick

> harti
> 
> -Original Message-
> From: Rick Macklem [mailto:rmack...@uoguelph.ca]
> Sent: Tuesday, May 14, 2013 2:50 PM
> To: Brandt, Hartmut
> Cc: curr...@freebsd.org
> Subject: Re: files disappearing from ls on NFS
> 
> Hartmut Brandt wrote:
> > On Mon, 13 May 2013, Rick Macklem wrote:
> >
> > RM>Hartmut Brandt wrote:
> > RM>> On Sun, 12 May 2013, Rick Macklem wrote:
> > RM>>
> > RM>> RM>Hartmut Brandt wrote:
> > RM>> RM>> Hi,
> > RM>> RM>>
> > RM>> RM>> I've updated one of my -current machines this week
> > (previous
> > RM>> update
> > RM>> RM>> was in
> > RM>> RM>> february). Now I see a strange effect (it seems only on
> > NFS
> > RM>> mounts):
> > RM>> RM>> ls or
> > RM>> RM>> even echo * will list only some files (strange enough the
> > first
> > RM>> files
> > RM>> RM>> from
> > RM>> RM>> the normal, alphabetically ordered list). If I change
> > something
> > RM>> in the
> > RM>> RM>> directory (delete a file or create a new one) for some
> > time
> > the
> > RM>> RM>> complete
> > RM>> RM>> listing will appear but after sime time (seconds to a
> > minute
> > or
> > RM>> so)
> > RM>> RM>> again
> > RM>> RM>> only part of the files is listed.
> > RM>> RM>>
> > RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> > getdirentries is
> > RM>> RM>> called
> > RM>> RM>> only once (returning 4096). For a full listing
> > getdirentries
> > is
> > RM>> called
> > RM>> RM>> 5
> > RM>> RM>> times with the last returning 0.
> > RM>> RM>>
> > RM>> RM>> I can still open files that are not listed if I know their
> > name,
> > RM>> RM>> though.
> > RM>> RM>>
> > RM>> RM>> The NFS server is a Windows 2008 server with an OpenText
> > NFS
> > RM>> Server
> > RM>> RM>> which
> > RM>> RM>> works without p

RE: files disappearing from ls on NFS

[Hartmut.Brandt]

Now I've also changed NFS_DIRBLKSIZ to 4k - no change.

harti

-Original Message-
From: Rick Macklem [mailto:rmack...@uoguelph.ca] 
Sent: Tuesday, May 14, 2013 2:50 PM
To: Brandt, Hartmut
Cc: curr...@freebsd.org
Subject: Re: files disappearing from ls on NFS

Hartmut Brandt wrote:
> On Mon, 13 May 2013, Rick Macklem wrote:
> 
> RM>Hartmut Brandt wrote:
> RM>> On Sun, 12 May 2013, Rick Macklem wrote:
> RM>>
> RM>> RM>Hartmut Brandt wrote:
> RM>> RM>> Hi,
> RM>> RM>>
> RM>> RM>> I've updated one of my -current machines this week (previous
> RM>> update
> RM>> RM>> was in
> RM>> RM>> february). Now I see a strange effect (it seems only on NFS
> RM>> mounts):
> RM>> RM>> ls or
> RM>> RM>> even echo * will list only some files (strange enough the
> first
> RM>> files
> RM>> RM>> from
> RM>> RM>> the normal, alphabetically ordered list). If I change
> something
> RM>> in the
> RM>> RM>> directory (delete a file or create a new one) for some time
> the
> RM>> RM>> complete
> RM>> RM>> listing will appear but after sime time (seconds to a minute
> or
> RM>> so)
> RM>> RM>> again
> RM>> RM>> only part of the files is listed.
> RM>> RM>>
> RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> getdirentries is
> RM>> RM>> called
> RM>> RM>> only once (returning 4096). For a full listing getdirentries
> is
> RM>> called
> RM>> RM>> 5
> RM>> RM>> times with the last returning 0.
> RM>> RM>>
> RM>> RM>> I can still open files that are not listed if I know their
> name,
> RM>> RM>> though.
> RM>> RM>>
> RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
> RM>> Server
> RM>> RM>> which
> RM>> RM>> works without problems to all the other FreeBSD machines.
> RM>> RM>>
> RM>> RM>> So what could that be?
> RM>> RM>>
> RM>> RM>I've attached a patch that might be worth trying. It is a
> "shot in
> RM>> the dark",
> RM>> RM>but brings the new NFS client's readdir closer to the old one
> RM>> (which you
> RM>> RM>mentioned still works ok).
> RM>> RM>
> RM>> RM>Please let me know how it goes, if you have a chance to test
> it,
> RM>> rick
> RM>>
> RM>> Hi Rick,
> RM>>
> RM>> the patch doesn't help.
> RM>>
> RM>> I wrote a small test program, which opens a directory, calls
> RM>> getdents(2)
> RM>> in a loop and dumps that. I figured out, that the return of the
> system
> RM>> call depends on the buffer size I pass to it. The directory has a 
> RM>> block size of 4k according to fstat(2). If I use that, I get some 
> RM>> 300
> of the
> RM>> almost 500 directory entries. If I use 8k, I get just around 200
> and
> RM>> if I
> RM>> use 16k I get a handfull. If I dump the buffer in this case I see
> RM>> 0x200
> RM>> bytes filled with directory entries, then a lot of zeros and
> starting
> RM>> from
> RM>> 0x1000 again data. This is of course ignored because of the zeros 
> RM>> before.
> RM>>
> RM>And for this case getdents(2) returned 16K? It is normal for
> getdents(2)
> RM>to return less than requested and when end of dir occurs, it should
> return 0.
> RM>
> RM>But if it returns 16K, there shouldn't be zeroed space in the
> middle of
> RM>it.
> RM>
> RM>And this always occurs or only after you wait a while? (You noted
> in the
> RM>above description that it would be ok for a little while after a
> directory
> RM>change and then would break, which suggests some kind of caching
> problem.)
> 
> Today in the morning everything was fine. After waiting 5 minutes, 
> again only partial directories. When I do a read with 8k buffer size,
> getdents(2) returns 8k, but starting from 0x200 until 0x1000 the 
> buffer is filled with zeros. The entry just before the zeroes ends 
> exactly at
> 0x200
> (that would be the first byte of the next entry) and at 0x1000 a new 
> entry starts. The rest of the buffer is fine. The next read returns 
> only 4k and seems to be fine - altough it contains some junk non-zero 
> bytes in the padding.
> 
Directory entries should never cross DIRBLKSIZ boundaries (512 or 0x200), so it 
ma

RE: files disappearing from ls on NFS

[Hartmut.Brandt]

Hi Rick,

sorry for top-posting - this is Outlook :-(

Attached is the system configuration. I use this more or less unchanged since 
years. The machine is an 8-core AMD64 with 144GByte memory.

The nfsstats -m output for the two file systems I'm testing with is:

knopfs01:/OP_UserUnix on /home
nfsv3,tcp,resvport,hard,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=6126856,timeout=120,retrans=2
knopfs01:/op_software on /software
nfsv3,tcp,resvport,hard,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=6126856,timeout=120,retrans=2

I did the tcpdump/wireshark thing and I'm puzzled that I see no readdir 
requests. I see a lookup, followed by getattr, access and fsstat for the 
directory and that's it. Looks that even after hours the stuff returned by 
getdirents(2) comes from the cache. I assume that the NFS client uses getattr 
to check whether
the directory has changed? If I knew what happens when calling getdirents() I 
could add some debugging printfs() here and there to figure out...

harti

-Original Message-
From: Rick Macklem [mailto:rmack...@uoguelph.ca] 
Sent: Tuesday, May 14, 2013 2:50 PM
To: Brandt, Hartmut
Cc: curr...@freebsd.org
Subject: Re: files disappearing from ls on NFS

Hartmut Brandt wrote:
> On Mon, 13 May 2013, Rick Macklem wrote:
> 
> RM>Hartmut Brandt wrote:
> RM>> On Sun, 12 May 2013, Rick Macklem wrote:
> RM>>
> RM>> RM>Hartmut Brandt wrote:
> RM>> RM>> Hi,
> RM>> RM>>
> RM>> RM>> I've updated one of my -current machines this week (previous
> RM>> update
> RM>> RM>> was in
> RM>> RM>> february). Now I see a strange effect (it seems only on NFS
> RM>> mounts):
> RM>> RM>> ls or
> RM>> RM>> even echo * will list only some files (strange enough the
> first
> RM>> files
> RM>> RM>> from
> RM>> RM>> the normal, alphabetically ordered list). If I change
> something
> RM>> in the
> RM>> RM>> directory (delete a file or create a new one) for some time
> the
> RM>> RM>> complete
> RM>> RM>> listing will appear but after sime time (seconds to a minute
> or
> RM>> so)
> RM>> RM>> again
> RM>> RM>> only part of the files is listed.
> RM>> RM>>
> RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> getdirentries is
> RM>> RM>> called
> RM>> RM>> only once (returning 4096). For a full listing getdirentries
> is
> RM>> called
> RM>> RM>> 5
> RM>> RM>> times with the last returning 0.
> RM>> RM>>
> RM>> RM>> I can still open files that are not listed if I know their
> name,
> RM>> RM>> though.
> RM>> RM>>
> RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
> RM>> Server
> RM>> RM>> which
> RM>> RM>> works without problems to all the other FreeBSD machines.
> RM>> RM>>
> RM>> RM>> So what could that be?
> RM>> RM>>
> RM>> RM>I've attached a patch that might be worth trying. It is a
> "shot in
> RM>> the dark",
> RM>> RM>but brings the new NFS client's readdir closer to the old one
> RM>> (which you
> RM>> RM>mentioned still works ok).
> RM>> RM>
> RM>> RM>Please let me know how it goes, if you have a chance to test
> it,
> RM>> rick
> RM>>
> RM>> Hi Rick,
> RM>>
> RM>> the patch doesn't help.
> RM>>
> RM>> I wrote a small test program, which opens a directory, calls
> RM>> getdents(2)
> RM>> in a loop and dumps that. I figured out, that the return of the
> system
> RM>> call depends on the buffer size I pass to it. The directory has a 
> RM>> block size of 4k according to fstat(2). If I use that, I get some 
> RM>> 300
> of the
> RM>> almost 500 directory entries. If I use 8k, I get just around 200
> and
> RM>> if I
> RM>> use 16k I get a handfull. If I dump the buffer in this case I see
> RM>> 0x200
> RM>> bytes filled with directory entries, then a lot of zeros and
> starting
> RM>> from
> RM>> 0x1000 again data. This is of course ignored because of the zeros 
> RM>> before.
> RM>>
> RM>And for this case getdents(2) returned 16K? It is normal for

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> On Mon, 13 May 2013, Rick Macklem wrote:
> 
> RM>Hartmut Brandt wrote:
> RM>> On Sun, 12 May 2013, Rick Macklem wrote:
> RM>>
> RM>> RM>Hartmut Brandt wrote:
> RM>> RM>> Hi,
> RM>> RM>>
> RM>> RM>> I've updated one of my -current machines this week (previous
> RM>> update
> RM>> RM>> was in
> RM>> RM>> february). Now I see a strange effect (it seems only on NFS
> RM>> mounts):
> RM>> RM>> ls or
> RM>> RM>> even echo * will list only some files (strange enough the
> first
> RM>> files
> RM>> RM>> from
> RM>> RM>> the normal, alphabetically ordered list). If I change
> something
> RM>> in the
> RM>> RM>> directory (delete a file or create a new one) for some time
> the
> RM>> RM>> complete
> RM>> RM>> listing will appear but after sime time (seconds to a minute
> or
> RM>> so)
> RM>> RM>> again
> RM>> RM>> only part of the files is listed.
> RM>> RM>>
> RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> getdirentries is
> RM>> RM>> called
> RM>> RM>> only once (returning 4096). For a full listing getdirentries
> is
> RM>> called
> RM>> RM>> 5
> RM>> RM>> times with the last returning 0.
> RM>> RM>>
> RM>> RM>> I can still open files that are not listed if I know their
> name,
> RM>> RM>> though.
> RM>> RM>>
> RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
> RM>> Server
> RM>> RM>> which
> RM>> RM>> works without problems to all the other FreeBSD machines.
> RM>> RM>>
> RM>> RM>> So what could that be?
> RM>> RM>>
> RM>> RM>I've attached a patch that might be worth trying. It is a
> "shot in
> RM>> the dark",
> RM>> RM>but brings the new NFS client's readdir closer to the old one
> RM>> (which you
> RM>> RM>mentioned still works ok).
> RM>> RM>
> RM>> RM>Please let me know how it goes, if you have a chance to test
> it,
> RM>> rick
> RM>>
> RM>> Hi Rick,
> RM>>
> RM>> the patch doesn't help.
> RM>>
> RM>> I wrote a small test program, which opens a directory, calls
> RM>> getdents(2)
> RM>> in a loop and dumps that. I figured out, that the return of the
> system
> RM>> call depends on the buffer size I pass to it. The directory has a
> RM>> block
> RM>> size of 4k according to fstat(2). If I use that, I get some 300
> of the
> RM>> almost 500 directory entries. If I use 8k, I get just around 200
> and
> RM>> if I
> RM>> use 16k I get a handfull. If I dump the buffer in this case I see
> RM>> 0x200
> RM>> bytes filled with directory entries, then a lot of zeros and
> starting
> RM>> from
> RM>> 0x1000 again data. This is of course ignored because of the zeros
> RM>> before.
> RM>>
> RM>And for this case getdents(2) returned 16K? It is normal for
> getdents(2)
> RM>to return less than requested and when end of dir occurs, it should
> return 0.
> RM>
> RM>But if it returns 16K, there shouldn't be zeroed space in the
> middle of
> RM>it.
> RM>
> RM>And this always occurs or only after you wait a while? (You noted
> in the
> RM>above description that it would be ok for a little while after a
> directory
> RM>change and then would break, which suggests some kind of caching
> problem.)
> 
> Today in the morning everything was fine. After waiting 5 minutes,
> again
> only partial directories. When I do a read with 8k buffer size,
> getdents(2) returns 8k, but starting from 0x200 until 0x1000 the
> buffer is
> filled with zeros. The entry just before the zeroes ends exactly at
> 0x200
> (that would be the first byte of the next entry) and at 0x1000 a new
> entry
> starts. The rest of the buffer is fine. The next read returns only 4k
> and
> seems to be fine - altough it contains some junk non-zero bytes in the
> padding.
> 
Directory entries should never cross DIRBLKSIZ boundaries (512 or 0x200),
so it makes sense that one ends at 0x200 and one starts at 0x1000. What
doesn't make sense are the 0 bytes in between.

One difference between the old and new NFS clients, which the patch I sent
you changed to the way the old one does it, is filling in the last block.
The old NFS client just leaves the block short and depends on n_direofoffset
to recognize it is the last block with b_resid indicating where it ends.
For the new client (unless you've applied the patch I emailed you), it fills
the rest of the last block in with "empty directories". This was in the OpenBSD
code when I did the original NFSv4 stuff and port. I left it in, because I
thought it might avoid problems if n_direofoffset was ever bogus. That is why
there might be "different junk" at the end of the directory, but it shouldn't
matter.

It almost sounds like something else is bzero()ing out part of the buffer
cache block. Unless the directory has changed, the getdents() after 5 minutes
would just return the same buffer cache block that was read in 5 minutes
earlier (unless the buffer fell out of the cache and had to be re-read from
the server, which would only happen if there was a lot of other file I/O
going on during that 5minutes).

A couple of comments:
- You can run "nfsstat -m" as root to see wha

Re: files disappearing from ls on NFS

[Hartmut Brandt]

On Mon, 13 May 2013, Rick Macklem wrote:

RM>Hartmut Brandt wrote:
RM>> On Sun, 12 May 2013, Rick Macklem wrote:
RM>> 
RM>> RM>Hartmut Brandt wrote:
RM>> RM>> Hi,
RM>> RM>>
RM>> RM>> I've updated one of my -current machines this week (previous
RM>> update
RM>> RM>> was in
RM>> RM>> february). Now I see a strange effect (it seems only on NFS
RM>> mounts):
RM>> RM>> ls or
RM>> RM>> even echo * will list only some files (strange enough the first
RM>> files
RM>> RM>> from
RM>> RM>> the normal, alphabetically ordered list). If I change something
RM>> in the
RM>> RM>> directory (delete a file or create a new one) for some time the
RM>> RM>> complete
RM>> RM>> listing will appear but after sime time (seconds to a minute or
RM>> so)
RM>> RM>> again
RM>> RM>> only part of the files is listed.
RM>> RM>>
RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
RM>> RM>> called
RM>> RM>> only once (returning 4096). For a full listing getdirentries is
RM>> called
RM>> RM>> 5
RM>> RM>> times with the last returning 0.
RM>> RM>>
RM>> RM>> I can still open files that are not listed if I know their name,
RM>> RM>> though.
RM>> RM>>
RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
RM>> Server
RM>> RM>> which
RM>> RM>> works without problems to all the other FreeBSD machines.
RM>> RM>>
RM>> RM>> So what could that be?
RM>> RM>>
RM>> RM>I've attached a patch that might be worth trying. It is a "shot in
RM>> the dark",
RM>> RM>but brings the new NFS client's readdir closer to the old one
RM>> (which you
RM>> RM>mentioned still works ok).
RM>> RM>
RM>> RM>Please let me know how it goes, if you have a chance to test it,
RM>> rick
RM>> 
RM>> Hi Rick,
RM>> 
RM>> the patch doesn't help.
RM>> 
RM>> I wrote a small test program, which opens a directory, calls
RM>> getdents(2)
RM>> in a loop and dumps that. I figured out, that the return of the system
RM>> call depends on the buffer size I pass to it. The directory has a
RM>> block
RM>> size of 4k according to fstat(2). If I use that, I get some 300 of the
RM>> almost 500 directory entries. If I use 8k, I get just around 200 and
RM>> if I
RM>> use 16k I get a handfull. If I dump the buffer in this case I see
RM>> 0x200
RM>> bytes filled with directory entries, then a lot of zeros and starting
RM>> from
RM>> 0x1000 again data. This is of course ignored because of the zeros
RM>> before.
RM>> 
RM>And for this case getdents(2) returned 16K? It is normal for getdents(2)
RM>to return less than requested and when end of dir occurs, it should return 0.
RM>
RM>But if it returns 16K, there shouldn't be zeroed space in the middle of
RM>it.
RM>
RM>And this always occurs or only after you wait a while? (You noted in the
RM>above description that it would be ok for a little while after a directory
RM>change and then would break, which suggests some kind of caching problem.)

Today in the morning everything was fine. After waiting 5 minutes, again 
only partial directories. When I do a read with 8k buffer size, 
getdents(2) returns 8k, but starting from 0x200 until 0x1000 the buffer is 
filled with zeros. The entry just before the zeroes ends exactly at 0x200 
(that would be the first byte of the next entry) and at 0x1000 a new entry 
starts. The rest of the buffer is fine. The next read returns only 4k and 
seems to be fine - altough it contains some junk non-zero bytes in the 
padding.

Ten minutes later again everything is fine. I tries to spy at the NFS 
communication with tcpdump, but it seems unwilling to display something 
useful about the NFS. Is it able to decode the readdir stuff?

harti

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> On Sun, 12 May 2013, Rick Macklem wrote:
> 
> RM>Hartmut Brandt wrote:
> RM>> Hi,
> RM>>
> RM>> I've updated one of my -current machines this week (previous
> update
> RM>> was in
> RM>> february). Now I see a strange effect (it seems only on NFS
> mounts):
> RM>> ls or
> RM>> even echo * will list only some files (strange enough the first
> files
> RM>> from
> RM>> the normal, alphabetically ordered list). If I change something
> in the
> RM>> directory (delete a file or create a new one) for some time the
> RM>> complete
> RM>> listing will appear but after sime time (seconds to a minute or
> so)
> RM>> again
> RM>> only part of the files is listed.
> RM>>
> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
> RM>> called
> RM>> only once (returning 4096). For a full listing getdirentries is
> called
> RM>> 5
> RM>> times with the last returning 0.
> RM>>
> RM>> I can still open files that are not listed if I know their name,
> RM>> though.
> RM>>
> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
> Server
> RM>> which
> RM>> works without problems to all the other FreeBSD machines.
> RM>>
> RM>> So what could that be?
> RM>>
> RM>I've attached a patch that might be worth trying. It is a "shot in
> the dark",
> RM>but brings the new NFS client's readdir closer to the old one
> (which you
> RM>mentioned still works ok).
> RM>
> RM>Please let me know how it goes, if you have a chance to test it,
> rick
> 
> Hi Rick,
> 
> the patch doesn't help.
> 
> I wrote a small test program, which opens a directory, calls
> getdents(2)
> in a loop and dumps that. I figured out, that the return of the system
> call depends on the buffer size I pass to it. The directory has a
> block
> size of 4k according to fstat(2). If I use that, I get some 300 of the
> almost 500 directory entries. If I use 8k, I get just around 200 and
> if I
> use 16k I get a handfull. If I dump the buffer in this case I see
> 0x200
> bytes filled with directory entries, then a lot of zeros and starting
> from
> 0x1000 again data. This is of course ignored because of the zeros
> before.
> 
And for this case getdents(2) returned 16K? It is normal for getdents(2)
to return less than requested and when end of dir occurs, it should return 0.

But if it returns 16K, there shouldn't be zeroed space in the middle of
it.

And this always occurs or only after you wait a while? (You noted in the
above description that it would be ok for a little while after a directory
change and then would break, which suggests some kind of caching problem.)

rick

> All this looks very strange.
> 
> harti
> 
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Hartmut Brandt]

On Sun, 12 May 2013, Rick Macklem wrote:

RM>Hartmut Brandt wrote:
RM>> Hi,
RM>> 
RM>> I've updated one of my -current machines this week (previous update
RM>> was in
RM>> february). Now I see a strange effect (it seems only on NFS mounts):
RM>> ls or
RM>> even echo * will list only some files (strange enough the first files
RM>> from
RM>> the normal, alphabetically ordered list). If I change something in the
RM>> directory (delete a file or create a new one) for some time the
RM>> complete
RM>> listing will appear but after sime time (seconds to a minute or so)
RM>> again
RM>> only part of the files is listed.
RM>> 
RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
RM>> called
RM>> only once (returning 4096). For a full listing getdirentries is called
RM>> 5
RM>> times with the last returning 0.
RM>> 
RM>> I can still open files that are not listed if I know their name,
RM>> though.
RM>> 
RM>> The NFS server is a Windows 2008 server with an OpenText NFS Server
RM>> which
RM>> works without problems to all the other FreeBSD machines.
RM>> 
RM>> So what could that be?
RM>> 
RM>I've attached a patch that might be worth trying. It is a "shot in the dark",
RM>but brings the new NFS client's readdir closer to the old one (which you
RM>mentioned still works ok).
RM>
RM>Please let me know how it goes, if you have a chance to test it, rick

Hi Rick,

the patch doesn't help.

I wrote a small test program, which opens a directory, calls getdents(2) 
in a loop and dumps that. I figured out, that the return of the system 
call depends on the buffer size I pass to it. The directory has a block 
size of 4k according to fstat(2). If I use that, I get some 300 of the 
almost 500 directory entries. If I use 8k, I get just around 200 and if I 
use 16k I get a handfull. If I dump the buffer in this case I see 0x200 
bytes filled with directory entries, then a lot of zeros and starting from 
0x1000 again data. This is of course ignored because of the zeros before.

All this looks very strange.

harti

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> Hi,
> 
> I've updated one of my -current machines this week (previous update
> was in
> february). Now I see a strange effect (it seems only on NFS mounts):
> ls or
> even echo * will list only some files (strange enough the first files
> from
> the normal, alphabetically ordered list). If I change something in the
> directory (delete a file or create a new one) for some time the
> complete
> listing will appear but after sime time (seconds to a minute or so)
> again
> only part of the files is listed.
> 
> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
> called
> only once (returning 4096). For a full listing getdirentries is called
> 5
> times with the last returning 0.
> 
> I can still open files that are not listed if I know their name,
> though.
> 
> The NFS server is a Windows 2008 server with an OpenText NFS Server
> which
> works without problems to all the other FreeBSD machines.
> 
> So what could that be?
> 
I've attached a patch that might be worth trying. It is a "shot in the dark",
but brings the new NFS client's readdir closer to the old one (which you
mentioned still works ok).

Please let me know how it goes, if you have a chance to test it, rick

> Regards,
> harti
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
--- fs/nfsclient/nfs_clrpcops.c.sav	2013-05-11 20:02:24.0 -0400
+++ fs/nfsclient/nfs_clrpcops.c	2013-05-11 20:46:28.0 -0400
@@ -2604,31 +2604,6 @@ nfsrpc_rmdir(vnode_t dvp, char *name, in
 
 /*
  * Readdir rpc.
- * Always returns with either uio_resid unchanged, if you are at the
- * end of the directory, or uio_resid == 0, with all DIRBLKSIZ chunks
- * filled in.
- * I felt this would allow caching of directory blocks more easily
- * than returning a pertially filled block.
- * Directory offset cookies:
- * Oh my, what to do with them...
- * I can think of three ways to deal with them:
- * 1 - have the layer above these RPCs maintain a map between logical
- * directory byte offsets and the NFS directory offset cookies
- * 2 - pass the opaque directory offset cookies up into userland
- * and let the libc functions deal with them, via the system call
- * 3 - return them to userland in the "struct dirent", so future versions
- * of libc can use them and do whatever is necessary to amke things work
- * above these rpc calls, in the meantime
- * For now, I do #3 by "hiding" the directory offset cookies after the
- * d_name field in struct dirent. This is space inside d_reclen that
- * will be ignored by anything that doesn't know about them.
- * The directory offset cookies are filled in as the last 8 bytes of
- * each directory entry, after d_name. Someday, the userland libc
- * functions may be able to use these. In the meantime, it satisfies
- * OpenBSD's requirements for cookies being returned.
- * If expects the directory offset cookie for the read to be in uio_offset
- * and returns the one for the next entry after this directory block in
- * there, as well.
  */
 APPLESTATIC int
 nfsrpc_readdir(vnode_t vp, struct uio *uiop, nfsuint64 *cookiep,
@@ -2649,12 +2624,12 @@ nfsrpc_readdir(vnode_t vp, struct uio *u
 	u_int32_t fakefileno = 0x, rderr;
 	char *cp;
 	nfsattrbit_t attrbits, dattrbits;
-	u_int32_t *tl2 = NULL;
-	size_t tresid;
+	ssize_t tresid;
 
 	KASSERT(uiop->uio_iovcnt == 1 &&
-	(uio_uio_resid(uiop) & (DIRBLKSIZ - 1)) == 0,
-	("nfs readdirrpc bad uio"));
+	(uiop->uio_offset & (DIRBLKSIZ - 1)) == 0 &&
+	(uiop->uio_resid & (DIRBLKSIZ - 1)) == 0,
+	("nfsrpc_readdir bad uio"));
 
 	/*
 	 * There is no point in reading a lot more than uio_resid, however
@@ -2663,13 +2638,13 @@ nfsrpc_readdir(vnode_t vp, struct uio *u
 	 * will never make readsize > nm_readdirsize.
 	 */
 	readsize = nmp->nm_readdirsize;
-	if (readsize > uio_uio_resid(uiop))
-		readsize = uio_uio_resid(uiop) + DIRBLKSIZ;
+	if (readsize > uiop->uio_resid)
+		readsize = uiop->uio_resid + DIRBLKSIZ;
 
 	*attrflagp = 0;
 	if (eofp)
 		*eofp = 0;
-	tresid = uio_uio_resid(uiop);
+	tresid = uiop->uio_resid;
 	cookie.lval[0] = cookiep->nfsuquad[0];
 	cookie.lval[1] = cookiep->nfsuquad[1];
 	nd->nd_mrep = NULL;
@@ -2765,43 +2740,33 @@ nfsrpc_readdir(vnode_t vp, struct uio *u
 			if (error)
 			return (error);
 			nd->nd_mrep = NULL;
-			dp = (struct dirent *) CAST_DOWN(caddr_t, uio_iov_base(uiop));
-			dp->d_type = DT_DIR;
+			dp = (struct dirent *)uiop->uio_iov->iov_base;
 			dp->d_fileno = dotfileid;
 			dp->d_namlen = 1;
 			dp->d_name[0] = '.';
 			dp->d_name[1] = '\0';
-			dp->d_reclen = DIRENT_SIZE(dp) + NFSX_HYPER;
-			/*
-			 * Just make these offset cookie 0.
-			 */
-			tl = (u_int32_t *)&dp->d_name[4];
-			*tl++ = 0;
-			*tl = 0;
+			dp->d_reclen = 4 + DIRHDSIZ;
+			dp->d_type = DT_DIR;
 			blksiz += dp->d_reclen;
-			uio_

Re: files disappearing from ls on NFS

[Hartmut Brandt]

On Mon, 6 May 2013, Rick Macklem wrote:

RM>Hartmut Brandt wrote:
RM>> Hi Rick,
RM>> 
RM>> the patch doesn't help. So how can I help to fix that? Of course, I
RM>> can use the work-around with oldnfs, but ...
RM>> 
RM>Well, I plan on going through the readdir code and seeing if I can spot
RM>a case that would break for small RPC replies. If I can find something,
RM>I'll email you a patch for testing. (I can't seem to reproduce the problem
RM>here.)
RM>
RM>The mysterious part for me is why it has shown up recently, because there
RM>hasn't been any recent change committed that seems like it could cause this.
RM>(Maybe it is just a co-incidence that it showed up recently and the bug has
RM> been there all along?)
RM>
RM>I'll admit my worst fear is that is somehow caused by the switch to clang for
RM>certain arches. If that is the case, it could take a long time to isolate.

I'm quite sure that I've build the system in February with clang already. 
But in march or so a new clang version was committed.

harti

RM>> -Original Message-
RM>> From: Rick Macklem [mailto:rmack...@uoguelph.ca]
RM>> Sent: Saturday, May 04, 2013 11:33 PM
RM>> To: Brandt, Hartmut
RM>> Cc: curr...@freebsd.org; Andrzej Tobola
RM>> Subject: Re: files disappearing from ls on NFS
RM>> 
RM>> Hartmut Brandt wrote:
RM>> > On Fri, 3 May 2013, Rick Macklem wrote:
RM>> >
RM>> > RM>Ok, if you succeed in isolating the commit, that would be great.
RM>> >
RM>> > Hmm. I'm somewhat stuck. clang from yesterday can't compile clang
RM>> > from
RM>> > a month ago...
RM>> >
RM>> > harti
RM>> >
RM>> Oh well. You could try this patch (which is the one to fix readdir for
RM>> union mounts), since I can see that VOP_VPTOCNP() will also be broken
RM>> without it. (I can't see how that would break "ls", but it breaks
RM>> __getcwd() and friends, so maybe it can affect "ls" somehow?)
RM>> 
RM>> It's a cut/paste under windows, so I'm afraid the whitespace will be
RM>> messed up, but it's pretty simple to apply by hand.
RM>> 
RM>> Index: nfs_clvnops.c
RM>> ===
RM>> --- nfs_clvnops.c (revision 249568)
RM>> +++ nfs_clvnops.c (working copy)
RM>> @@ -2221,6 +2221,7 @@
RM>> !NFS_TIMESPEC_COMPARE(&np->n_mtime, &vattr.va_mtime)) {
RM>> mtx_unlock(&np->n_mtx);
RM>> NFSINCRGLOBAL(newnfsstats.direofcache_hits);
RM>> + *ap->a_eofflag = 1;
RM>> return (0);
RM>> } else
RM>> mtx_unlock(&np->n_mtx); @@ -2233,8 +2234,10 @@
RM>> tresid = uio->uio_resid;
RM>> error = ncl_bioread(vp, uio, 0, ap->a_cred);
RM>> 
RM>> - if (!error && uio->uio_resid == tresid)
RM>> + if (!error && uio->uio_resid == tresid) {
RM>> NFSINCRGLOBAL(newnfsstats.direofcache_misses);
RM>> + *ap->a_eofflag = 1;
RM>> + }
RM>> return (error);
RM>> }
RM>> 
RM>> I haven't yet succeeded in reproducing the problem, but will be poking
RM>> at it some more, rick
RM>> 
RM>> > RM>
RM>> > RM>rick
RM>> > RM>
RM>> > RM>> harti
RM>> > RM>>
RM>> > RM>> On Fri, 3 May 2013, Rick Macklem wrote:
RM>> > RM>>
RM>> > RM>> RM>Hartmut Brandt wrote:
RM>> > RM>> RM>> Hi,
RM>> > RM>> RM>>
RM>> > RM>> RM>> I've updated one of my -current machines this week
RM>> > (previous
RM>> > RM>> update
RM>> > RM>> RM>> was in
RM>> > RM>> RM>> february). Now I see a strange effect (it seems only on
RM>> > NFS
RM>> > RM>> mounts):
RM>> > RM>> RM>> ls or
RM>> > RM>> RM>> even echo * will list only some files (strange enough the
RM>> > first
RM>> > RM>> files
RM>> > RM>> RM>> from
RM>> > RM>> RM>> the normal, alphabetically ordered list). If I change
RM>> > something
RM>> > RM>> in the
RM>> > RM>> RM>> directory (delete a file or create a new one) for some
RM>> > time
RM>> > the
RM>> > RM>> RM>> complete
RM>> > RM>> RM>> listing will appear but after sime time (seconds to a
RM>> > minute
RM>> > or
RM>> > RM>> so)
RM>> > RM>> RM>> again
RM>> > RM>> RM>> only part of the files is li

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> Hi Rick,
> 
> the patch doesn't help. So how can I help to fix that? Of course, I
> can use the work-around with oldnfs, but ...
> 
Well, I plan on going through the readdir code and seeing if I can spot
a case that would break for small RPC replies. If I can find something,
I'll email you a patch for testing. (I can't seem to reproduce the problem
here.)

The mysterious part for me is why it has shown up recently, because there
hasn't been any recent change committed that seems like it could cause this.
(Maybe it is just a co-incidence that it showed up recently and the bug has
 been there all along?)

I'll admit my worst fear is that is somehow caused by the switch to clang for
certain arches. If that is the case, it could take a long time to isolate.

rick
> harti
> 
> -Original Message-
> From: Rick Macklem [mailto:rmack...@uoguelph.ca]
> Sent: Saturday, May 04, 2013 11:33 PM
> To: Brandt, Hartmut
> Cc: curr...@freebsd.org; Andrzej Tobola
> Subject: Re: files disappearing from ls on NFS
> 
> Hartmut Brandt wrote:
> > On Fri, 3 May 2013, Rick Macklem wrote:
> >
> > RM>Ok, if you succeed in isolating the commit, that would be great.
> >
> > Hmm. I'm somewhat stuck. clang from yesterday can't compile clang
> > from
> > a month ago...
> >
> > harti
> >
> Oh well. You could try this patch (which is the one to fix readdir for
> union mounts), since I can see that VOP_VPTOCNP() will also be broken
> without it. (I can't see how that would break "ls", but it breaks
> __getcwd() and friends, so maybe it can affect "ls" somehow?)
> 
> It's a cut/paste under windows, so I'm afraid the whitespace will be
> messed up, but it's pretty simple to apply by hand.
> 
> Index: nfs_clvnops.c
> ===
> --- nfs_clvnops.c (revision 249568)
> +++ nfs_clvnops.c (working copy)
> @@ -2221,6 +2221,7 @@
> !NFS_TIMESPEC_COMPARE(&np->n_mtime, &vattr.va_mtime)) {
> mtx_unlock(&np->n_mtx);
> NFSINCRGLOBAL(newnfsstats.direofcache_hits);
> + *ap->a_eofflag = 1;
> return (0);
> } else
> mtx_unlock(&np->n_mtx); @@ -2233,8 +2234,10 @@
> tresid = uio->uio_resid;
> error = ncl_bioread(vp, uio, 0, ap->a_cred);
> 
> - if (!error && uio->uio_resid == tresid)
> + if (!error && uio->uio_resid == tresid) {
> NFSINCRGLOBAL(newnfsstats.direofcache_misses);
> + *ap->a_eofflag = 1;
> + }
> return (error);
> }
> 
> I haven't yet succeeded in reproducing the problem, but will be poking
> at it some more, rick
> 
> > RM>
> > RM>rick
> > RM>
> > RM>> harti
> > RM>>
> > RM>> On Fri, 3 May 2013, Rick Macklem wrote:
> > RM>>
> > RM>> RM>Hartmut Brandt wrote:
> > RM>> RM>> Hi,
> > RM>> RM>>
> > RM>> RM>> I've updated one of my -current machines this week
> > (previous
> > RM>> update
> > RM>> RM>> was in
> > RM>> RM>> february). Now I see a strange effect (it seems only on
> > NFS
> > RM>> mounts):
> > RM>> RM>> ls or
> > RM>> RM>> even echo * will list only some files (strange enough the
> > first
> > RM>> files
> > RM>> RM>> from
> > RM>> RM>> the normal, alphabetically ordered list). If I change
> > something
> > RM>> in the
> > RM>> RM>> directory (delete a file or create a new one) for some
> > time
> > the
> > RM>> RM>> complete
> > RM>> RM>> listing will appear but after sime time (seconds to a
> > minute
> > or
> > RM>> so)
> > RM>> RM>> again
> > RM>> RM>> only part of the files is listed.
> > RM>> RM>>
> > RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> > getdirentries is
> > RM>> RM>> called
> > RM>> RM>> only once (returning 4096). For a full listing
> > getdirentries
> > is
> > RM>> called
> > RM>> RM>> 5
> > RM>> RM>> times with the last returning 0.
> > RM>> RM>>
> > RM>> RM>> I can still open files that are not listed if I know their
> > name,
> > RM>> RM>> though.
> > RM>> RM>>
> > RM>> RM>> The NFS server is a Windows 2008 server with an OpenText
> > NFS
> > RM>> Server
> > RM>> RM>> 

RE: files disappearing from ls on NFS

[Hartmut.Brandt]

Hi Rick,

the patch doesn't help. So how can I help to fix that? Of course, I can use the 
work-around with oldnfs, but ...

harti

-Original Message-
From: Rick Macklem [mailto:rmack...@uoguelph.ca] 
Sent: Saturday, May 04, 2013 11:33 PM
To: Brandt, Hartmut
Cc: curr...@freebsd.org; Andrzej Tobola
Subject: Re: files disappearing from ls on NFS

Hartmut Brandt wrote:
> On Fri, 3 May 2013, Rick Macklem wrote:
> 
> RM>Ok, if you succeed in isolating the commit, that would be great.
> 
> Hmm. I'm somewhat stuck. clang from yesterday can't compile clang from 
> a month ago...
> 
> harti
> 
Oh well. You could try this patch (which is the one to fix readdir for union 
mounts), since I can see that VOP_VPTOCNP() will also be broken without it. (I 
can't see how that would break "ls", but it breaks __getcwd() and friends, so 
maybe it can affect "ls" somehow?)

It's a cut/paste under windows, so I'm afraid the whitespace will be messed up, 
but it's pretty simple to apply by hand.

Index: nfs_clvnops.c
===
--- nfs_clvnops.c(revision 249568)
+++ nfs_clvnops.c(working copy)
@@ -2221,6 +2221,7 @@
 !NFS_TIMESPEC_COMPARE(&np->n_mtime, 
&vattr.va_mtime)) {
 mtx_unlock(&np->n_mtx);
 NFSINCRGLOBAL(newnfsstats.direofcache_hits);
+*ap->a_eofflag = 1;
 return (0);
 } else
 mtx_unlock(&np->n_mtx); @@ -2233,8 +2234,10 @@
 tresid = uio->uio_resid;
 error = ncl_bioread(vp, uio, 0, ap->a_cred);
 
-if (!error && uio->uio_resid == tresid)
+if (!error && uio->uio_resid == tresid) {
 NFSINCRGLOBAL(newnfsstats.direofcache_misses);
+*ap->a_eofflag = 1;
+}
 return (error);
 }

I haven't yet succeeded in reproducing the problem, but will be poking at it 
some more, rick

> RM>
> RM>rick
> RM>
> RM>> harti
> RM>>
> RM>> On Fri, 3 May 2013, Rick Macklem wrote:
> RM>>
> RM>> RM>Hartmut Brandt wrote:
> RM>> RM>> Hi,
> RM>> RM>>
> RM>> RM>> I've updated one of my -current machines this week (previous
> RM>> update
> RM>> RM>> was in
> RM>> RM>> february). Now I see a strange effect (it seems only on NFS
> RM>> mounts):
> RM>> RM>> ls or
> RM>> RM>> even echo * will list only some files (strange enough the
> first
> RM>> files
> RM>> RM>> from
> RM>> RM>> the normal, alphabetically ordered list). If I change
> something
> RM>> in the
> RM>> RM>> directory (delete a file or create a new one) for some time
> the
> RM>> RM>> complete
> RM>> RM>> listing will appear but after sime time (seconds to a minute
> or
> RM>> so)
> RM>> RM>> again
> RM>> RM>> only part of the files is listed.
> RM>> RM>>
> RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> getdirentries is
> RM>> RM>> called
> RM>> RM>> only once (returning 4096). For a full listing getdirentries
> is
> RM>> called
> RM>> RM>> 5
> RM>> RM>> times with the last returning 0.
> RM>> RM>>
> RM>> RM>> I can still open files that are not listed if I know their
> name,
> RM>> RM>> though.
> RM>> RM>>
> RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
> RM>> Server
> RM>> RM>> which
> RM>> RM>> works without problems to all the other FreeBSD machines.
> RM>> RM>>
> RM>> RM>> So what could that be?
> RM>> RM>>
> RM>> RM>Someone else reported missing files returned via "ls"
> recently,
> RM>> when
> RM>> RM>they used a small readdirsize (below 8K). I haven't yet had a
> RM>> change to try
> RM>> RM>and reproduce it or do any snooping around.
> RM>> RM>
> RM>> RM>There haven't been any recent changes to readdir in the NFS
> client,
> RM>> RM>except a trivial one that adds a check for vnode type being
> VDIR,
> RM>> RM>so I don't see that it can be a recent NFS change.
> RM>> RM>
> RM>> RM>If you can increase the readdirsize, try that to see if

RE: files disappearing from ls on NFS

[Hartmut.Brandt]

Looks like the problem is in the new NFS code - the old code does the right 
thing. I've still to try your patch...

harti

From: Rick Macklem [rmack...@uoguelph.ca]
Sent: Sunday, May 05, 2013 12:49 AM
To: Brandt, Hartmut
Cc: curr...@freebsd.org
Subject: Re: files disappearing from ls on NFS

Hartmut Brandt wrote:
> On Fri, 3 May 2013, Rick Macklem wrote:
>
> RM>Ok, if you succeed in isolating the commit, that would be great.
>
> Hmm. I'm somewhat stuck. clang from yesterday can't compile clang from
> a
> month ago...
>
> harti
>
Oh, and one other thing you can try is switching to the old client
"mount -t oldnfs ...".

rick

> RM>
> RM>rick
> RM>
> RM>> harti
> RM>>
> RM>> On Fri, 3 May 2013, Rick Macklem wrote:
> RM>>
> RM>> RM>Hartmut Brandt wrote:
> RM>> RM>> Hi,
> RM>> RM>>
> RM>> RM>> I've updated one of my -current machines this week (previous
> RM>> update
> RM>> RM>> was in
> RM>> RM>> february). Now I see a strange effect (it seems only on NFS
> RM>> mounts):
> RM>> RM>> ls or
> RM>> RM>> even echo * will list only some files (strange enough the
> first
> RM>> files
> RM>> RM>> from
> RM>> RM>> the normal, alphabetically ordered list). If I change
> something
> RM>> in the
> RM>> RM>> directory (delete a file or create a new one) for some time
> the
> RM>> RM>> complete
> RM>> RM>> listing will appear but after sime time (seconds to a minute
> or
> RM>> so)
> RM>> RM>> again
> RM>> RM>> only part of the files is listed.
> RM>> RM>>
> RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> getdirentries is
> RM>> RM>> called
> RM>> RM>> only once (returning 4096). For a full listing getdirentries
> is
> RM>> called
> RM>> RM>> 5
> RM>> RM>> times with the last returning 0.
> RM>> RM>>
> RM>> RM>> I can still open files that are not listed if I know their
> name,
> RM>> RM>> though.
> RM>> RM>>
> RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
> RM>> Server
> RM>> RM>> which
> RM>> RM>> works without problems to all the other FreeBSD machines.
> RM>> RM>>
> RM>> RM>> So what could that be?
> RM>> RM>>
> RM>> RM>Someone else reported missing files returned via "ls"
> recently,
> RM>> when
> RM>> RM>they used a small readdirsize (below 8K). I haven't yet had a
> RM>> change to try
> RM>> RM>and reproduce it or do any snooping around.
> RM>> RM>
> RM>> RM>There haven't been any recent changes to readdir in the NFS
> client,
> RM>> RM>except a trivial one that adds a check for vnode type being
> VDIR,
> RM>> RM>so I don't see that it can be a recent NFS change.
> RM>> RM>
> RM>> RM>If you can increase the readdirsize, try that to see if it
> avoids
> RM>> RM>the problem. "nfsstat -m" shows you what the mount options end
> up
> RM>> RM>being after doing the mount. The server might be limiting the
> RM>> readdirsize
> RM>> RM>to 4K, so you should check, even if you specify a large value
> for
> RM>> RM>the mount.
> RM>> RM>
> RM>> RM>rick
> RM>> RM>
> RM>> RM>> Regards,
> RM>> RM>> harti
> RM>> RM>> ___
> RM>> RM>> freebsd-current@freebsd.org mailing list
> RM>> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> RM>> RM>> To unsubscribe, send any mail to
> RM>> RM>> "freebsd-current-unsubscr...@freebsd.org"
> RM>> RM>
> RM>> ___
> RM>> freebsd-current@freebsd.org mailing list
> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> RM>> To unsubscribe, send any mail to
> RM>> "freebsd-current-unsubscr...@freebsd.org"
> RM>
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> On Fri, 3 May 2013, Rick Macklem wrote:
> 
> RM>Ok, if you succeed in isolating the commit, that would be great.
> 
> Hmm. I'm somewhat stuck. clang from yesterday can't compile clang from
> a
> month ago...
> 
> harti
> 
Oh, and one other thing you can try is switching to the old client
"mount -t oldnfs ...".

rick

> RM>
> RM>rick
> RM>
> RM>> harti
> RM>>
> RM>> On Fri, 3 May 2013, Rick Macklem wrote:
> RM>>
> RM>> RM>Hartmut Brandt wrote:
> RM>> RM>> Hi,
> RM>> RM>>
> RM>> RM>> I've updated one of my -current machines this week (previous
> RM>> update
> RM>> RM>> was in
> RM>> RM>> february). Now I see a strange effect (it seems only on NFS
> RM>> mounts):
> RM>> RM>> ls or
> RM>> RM>> even echo * will list only some files (strange enough the
> first
> RM>> files
> RM>> RM>> from
> RM>> RM>> the normal, alphabetically ordered list). If I change
> something
> RM>> in the
> RM>> RM>> directory (delete a file or create a new one) for some time
> the
> RM>> RM>> complete
> RM>> RM>> listing will appear but after sime time (seconds to a minute
> or
> RM>> so)
> RM>> RM>> again
> RM>> RM>> only part of the files is listed.
> RM>> RM>>
> RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> getdirentries is
> RM>> RM>> called
> RM>> RM>> only once (returning 4096). For a full listing getdirentries
> is
> RM>> called
> RM>> RM>> 5
> RM>> RM>> times with the last returning 0.
> RM>> RM>>
> RM>> RM>> I can still open files that are not listed if I know their
> name,
> RM>> RM>> though.
> RM>> RM>>
> RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
> RM>> Server
> RM>> RM>> which
> RM>> RM>> works without problems to all the other FreeBSD machines.
> RM>> RM>>
> RM>> RM>> So what could that be?
> RM>> RM>>
> RM>> RM>Someone else reported missing files returned via "ls"
> recently,
> RM>> when
> RM>> RM>they used a small readdirsize (below 8K). I haven't yet had a
> RM>> change to try
> RM>> RM>and reproduce it or do any snooping around.
> RM>> RM>
> RM>> RM>There haven't been any recent changes to readdir in the NFS
> client,
> RM>> RM>except a trivial one that adds a check for vnode type being
> VDIR,
> RM>> RM>so I don't see that it can be a recent NFS change.
> RM>> RM>
> RM>> RM>If you can increase the readdirsize, try that to see if it
> avoids
> RM>> RM>the problem. "nfsstat -m" shows you what the mount options end
> up
> RM>> RM>being after doing the mount. The server might be limiting the
> RM>> readdirsize
> RM>> RM>to 4K, so you should check, even if you specify a large value
> for
> RM>> RM>the mount.
> RM>> RM>
> RM>> RM>rick
> RM>> RM>
> RM>> RM>> Regards,
> RM>> RM>> harti
> RM>> RM>> ___
> RM>> RM>> freebsd-current@freebsd.org mailing list
> RM>> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> RM>> RM>> To unsubscribe, send any mail to
> RM>> RM>> "freebsd-current-unsubscr...@freebsd.org"
> RM>> RM>
> RM>> ___
> RM>> freebsd-current@freebsd.org mailing list
> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> RM>> To unsubscribe, send any mail to
> RM>> "freebsd-current-unsubscr...@freebsd.org"
> RM>
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> On Fri, 3 May 2013, Rick Macklem wrote:
> 
> RM>Ok, if you succeed in isolating the commit, that would be great.
> 
> Hmm. I'm somewhat stuck. clang from yesterday can't compile clang from
> a
> month ago...
> 
> harti
> 
Oh well. You could try this patch (which is the one to fix readdir for
union mounts), since I can see that VOP_VPTOCNP() will also be broken
without it. (I can't see how that would break "ls", but it breaks __getcwd()
and friends, so maybe it can affect "ls" somehow?)

It's a cut/paste under windows, so I'm afraid the whitespace will be
messed up, but it's pretty simple to apply by hand.

Index: nfs_clvnops.c
===
--- nfs_clvnops.c(revision 249568)
+++ nfs_clvnops.c(working copy)
@@ -2221,6 +2221,7 @@
 !NFS_TIMESPEC_COMPARE(&np->n_mtime, 
&vattr.va_mtime)) {
 mtx_unlock(&np->n_mtx);
 NFSINCRGLOBAL(newnfsstats.direofcache_hits);
+*ap->a_eofflag = 1;
 return (0);
 } else
 mtx_unlock(&np->n_mtx);
@@ -2233,8 +2234,10 @@
 tresid = uio->uio_resid;
 error = ncl_bioread(vp, uio, 0, ap->a_cred);
 
-if (!error && uio->uio_resid == tresid)
+if (!error && uio->uio_resid == tresid) {
 NFSINCRGLOBAL(newnfsstats.direofcache_misses);
+*ap->a_eofflag = 1;
+}
 return (error);
 }

I haven't yet succeeded in reproducing the problem, but will be poking
at it some more, rick

> RM>
> RM>rick
> RM>
> RM>> harti
> RM>>
> RM>> On Fri, 3 May 2013, Rick Macklem wrote:
> RM>>
> RM>> RM>Hartmut Brandt wrote:
> RM>> RM>> Hi,
> RM>> RM>>
> RM>> RM>> I've updated one of my -current machines this week (previous
> RM>> update
> RM>> RM>> was in
> RM>> RM>> february). Now I see a strange effect (it seems only on NFS
> RM>> mounts):
> RM>> RM>> ls or
> RM>> RM>> even echo * will list only some files (strange enough the
> first
> RM>> files
> RM>> RM>> from
> RM>> RM>> the normal, alphabetically ordered list). If I change
> something
> RM>> in the
> RM>> RM>> directory (delete a file or create a new one) for some time
> the
> RM>> RM>> complete
> RM>> RM>> listing will appear but after sime time (seconds to a minute
> or
> RM>> so)
> RM>> RM>> again
> RM>> RM>> only part of the files is listed.
> RM>> RM>>
> RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that
> getdirentries is
> RM>> RM>> called
> RM>> RM>> only once (returning 4096). For a full listing getdirentries
> is
> RM>> called
> RM>> RM>> 5
> RM>> RM>> times with the last returning 0.
> RM>> RM>>
> RM>> RM>> I can still open files that are not listed if I know their
> name,
> RM>> RM>> though.
> RM>> RM>>
> RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
> RM>> Server
> RM>> RM>> which
> RM>> RM>> works without problems to all the other FreeBSD machines.
> RM>> RM>>
> RM>> RM>> So what could that be?
> RM>> RM>>
> RM>> RM>Someone else reported missing files returned via "ls"
> recently,
> RM>> when
> RM>> RM>they used a small readdirsize (below 8K). I haven't yet had a
> RM>> change to try
> RM>> RM>and reproduce it or do any snooping around.
> RM>> RM>
> RM>> RM>There haven't been any recent changes to readdir in the NFS
> client,
> RM>> RM>except a trivial one that adds a check for vnode type being
> VDIR,
> RM>> RM>so I don't see that it can be a recent NFS change.
> RM>> RM>
> RM>> RM>If you can increase the readdirsize, try that to see if it
> avoids
> RM>> RM>the problem. "nfsstat -m" shows you what the mount options end
> up
> RM>> RM>being after doing the mount. The server might be limiting the
> RM>> readdirsize
> RM>> RM>to 4K, so you should check, even if you specify a large value
> for
> RM>> RM>the mount.
> RM>> RM>
> RM>> RM>rick
> RM>> RM>
> RM>> RM>> Regards,
> RM>> RM>> harti
> RM>> RM>> ___
> RM>> RM>> freebsd-current@freebsd.org mailing list
> RM>> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> RM>> RM>> To unsubscribe, send any mail to
> RM>> RM>> "freebsd-current-unsubscr...@freebsd.org"
> RM>> RM>
> RM>> ___
> RM>> freebsd-current@freebsd.org mailing list
> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> RM>> To unsubscribe, send any mail to
> RM>> "freebsd-current-unsubscr...@freebsd.org"
> RM>
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to

Re: files disappearing from ls on NFS

[Hartmut Brandt]

On Fri, 3 May 2013, Rick Macklem wrote:

RM>Ok, if you succeed in isolating the commit, that would be great.

Hmm. I'm somewhat stuck. clang from yesterday can't compile clang from a 
month ago...

harti

RM>
RM>rick
RM>
RM>> harti
RM>> 
RM>> On Fri, 3 May 2013, Rick Macklem wrote:
RM>> 
RM>> RM>Hartmut Brandt wrote:
RM>> RM>> Hi,
RM>> RM>>
RM>> RM>> I've updated one of my -current machines this week (previous
RM>> update
RM>> RM>> was in
RM>> RM>> february). Now I see a strange effect (it seems only on NFS
RM>> mounts):
RM>> RM>> ls or
RM>> RM>> even echo * will list only some files (strange enough the first
RM>> files
RM>> RM>> from
RM>> RM>> the normal, alphabetically ordered list). If I change something
RM>> in the
RM>> RM>> directory (delete a file or create a new one) for some time the
RM>> RM>> complete
RM>> RM>> listing will appear but after sime time (seconds to a minute or
RM>> so)
RM>> RM>> again
RM>> RM>> only part of the files is listed.
RM>> RM>>
RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
RM>> RM>> called
RM>> RM>> only once (returning 4096). For a full listing getdirentries is
RM>> called
RM>> RM>> 5
RM>> RM>> times with the last returning 0.
RM>> RM>>
RM>> RM>> I can still open files that are not listed if I know their name,
RM>> RM>> though.
RM>> RM>>
RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
RM>> Server
RM>> RM>> which
RM>> RM>> works without problems to all the other FreeBSD machines.
RM>> RM>>
RM>> RM>> So what could that be?
RM>> RM>>
RM>> RM>Someone else reported missing files returned via "ls" recently,
RM>> when
RM>> RM>they used a small readdirsize (below 8K). I haven't yet had a
RM>> change to try
RM>> RM>and reproduce it or do any snooping around.
RM>> RM>
RM>> RM>There haven't been any recent changes to readdir in the NFS client,
RM>> RM>except a trivial one that adds a check for vnode type being VDIR,
RM>> RM>so I don't see that it can be a recent NFS change.
RM>> RM>
RM>> RM>If you can increase the readdirsize, try that to see if it avoids
RM>> RM>the problem. "nfsstat -m" shows you what the mount options end up
RM>> RM>being after doing the mount. The server might be limiting the
RM>> readdirsize
RM>> RM>to 4K, so you should check, even if you specify a large value for
RM>> RM>the mount.
RM>> RM>
RM>> RM>rick
RM>> RM>
RM>> RM>> Regards,
RM>> RM>> harti
RM>> RM>> ___
RM>> RM>> freebsd-current@freebsd.org mailing list
RM>> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
RM>> RM>> To unsubscribe, send any mail to
RM>> RM>> "freebsd-current-unsubscr...@freebsd.org"
RM>> RM>
RM>> ___
RM>> freebsd-current@freebsd.org mailing list
RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
RM>> To unsubscribe, send any mail to
RM>> "freebsd-current-unsubscr...@freebsd.org"
RM>
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Daniel Braniss]

> Hartmut Brandt wrote:
> > On Fri, 3 May 2013, Daniel Braniss wrote:
> > 
> > DB>I don't know about current, but on 9.1-stable, the nfsstat -m only
> > works
> > DB>for root! nfsstat can be run by anybody.
> > 
> > Same for current. It silently prints nothing. Took me some time
> > to figure out I should try as root...
> > 
> Yea, I suppose it should either be opened up to non-root (see previous post)
> or "only works as root" should be documented.
> 
I have no strong opinios either, though security by obscurity was never my 
favorit.
It should at least have said 'permition denied'.

cheers,
danny

> rick
> 
> > harti


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> Hi,
> 
> I've updated one of my -current machines this week (previous update
> was in
> february). Now I see a strange effect (it seems only on NFS mounts):
> ls or
> even echo * will list only some files (strange enough the first files
> from
> the normal, alphabetically ordered list). If I change something in the
> directory (delete a file or create a new one) for some time the
> complete
> listing will appear but after sime time (seconds to a minute or so)
> again
> only part of the files is listed.
> 
> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
> called
> only once (returning 4096). For a full listing getdirentries is called
> 5
> times with the last returning 0.
> 
> I can still open files that are not listed if I know their name,
> though.
> 
> The NFS server is a Windows 2008 server with an OpenText NFS Server
> which
> works without problems to all the other FreeBSD machines.
> 
> So what could that be?
> 
Just in case..is the NFS mount in a union mount by any chance?
(There is a known bug for NFS readdir under a union mount, because the
 NFS client readdir isn't setting the eofflag. Someone has sent me a patch
 that looks fine, but it hasn't been checked into head yet.)

I'll also grep a head kernel to check to see if anything else (except the
NFS server) uses the eofflag argument to VOP_READDIR().

rick

> Regards,
> harti
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> Hi Rick,
> 
> I checked. readdirsize is 64k.
> 
Since the server replied with 4K, I suspect you are seeing the same
problem the other reporter has.

I'll email if/when I have more information on it.

> I will try to do a binary search for the problematic commit next week.
> For
> this I had to do a local checkout (usually I have the system sources
> on
> the file server and this fails, of course).
> 
Ok, if you succeed in isolating the commit, that would be great.

rick

> harti
> 
> On Fri, 3 May 2013, Rick Macklem wrote:
> 
> RM>Hartmut Brandt wrote:
> RM>> Hi,
> RM>>
> RM>> I've updated one of my -current machines this week (previous
> update
> RM>> was in
> RM>> february). Now I see a strange effect (it seems only on NFS
> mounts):
> RM>> ls or
> RM>> even echo * will list only some files (strange enough the first
> files
> RM>> from
> RM>> the normal, alphabetically ordered list). If I change something
> in the
> RM>> directory (delete a file or create a new one) for some time the
> RM>> complete
> RM>> listing will appear but after sime time (seconds to a minute or
> so)
> RM>> again
> RM>> only part of the files is listed.
> RM>>
> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
> RM>> called
> RM>> only once (returning 4096). For a full listing getdirentries is
> called
> RM>> 5
> RM>> times with the last returning 0.
> RM>>
> RM>> I can still open files that are not listed if I know their name,
> RM>> though.
> RM>>
> RM>> The NFS server is a Windows 2008 server with an OpenText NFS
> Server
> RM>> which
> RM>> works without problems to all the other FreeBSD machines.
> RM>>
> RM>> So what could that be?
> RM>>
> RM>Someone else reported missing files returned via "ls" recently,
> when
> RM>they used a small readdirsize (below 8K). I haven't yet had a
> change to try
> RM>and reproduce it or do any snooping around.
> RM>
> RM>There haven't been any recent changes to readdir in the NFS client,
> RM>except a trivial one that adds a check for vnode type being VDIR,
> RM>so I don't see that it can be a recent NFS change.
> RM>
> RM>If you can increase the readdirsize, try that to see if it avoids
> RM>the problem. "nfsstat -m" shows you what the mount options end up
> RM>being after doing the mount. The server might be limiting the
> readdirsize
> RM>to 4K, so you should check, even if you specify a large value for
> RM>the mount.
> RM>
> RM>rick
> RM>
> RM>> Regards,
> RM>> harti
> RM>> ___
> RM>> freebsd-current@freebsd.org mailing list
> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> RM>> To unsubscribe, send any mail to
> RM>> "freebsd-current-unsubscr...@freebsd.org"
> RM>
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> On Fri, 3 May 2013, Daniel Braniss wrote:
> 
> DB>I don't know about current, but on 9.1-stable, the nfsstat -m only
> works
> DB>for root! nfsstat can be run by anybody.
> 
> Same for current. It silently prints nothing. Took me some time
> to figure out I should try as root...
> 
Yea, I suppose it should either be opened up to non-root (see previous post)
or "only works as root" should be documented.

rick

> harti
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Rick Macklem]

Daniel Braniss wrote:
> > Hartmut Brandt wrote:
> > > Hi,
> > >
> > > I've updated one of my -current machines this week (previous
> > > update
> > > was in
> > > february). Now I see a strange effect (it seems only on NFS
> > > mounts):
> > > ls or
> > > even echo * will list only some files (strange enough the first
> > > files
> > > from
> > > the normal, alphabetically ordered list). If I change something in
> > > the
> > > directory (delete a file or create a new one) for some time the
> > > complete
> > > listing will appear but after sime time (seconds to a minute or
> > > so)
> > > again
> > > only part of the files is listed.
> > >
> > > A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
> > > called
> > > only once (returning 4096). For a full listing getdirentries is
> > > called
> > > 5
> > > times with the last returning 0.
> > >
> > > I can still open files that are not listed if I know their name,
> > > though.
> > >
> > > The NFS server is a Windows 2008 server with an OpenText NFS
> > > Server
> > > which
> > > works without problems to all the other FreeBSD machines.
> > >
> > > So what could that be?
> > >
> > Someone else reported missing files returned via "ls" recently, when
> > they used a small readdirsize (below 8K). I haven't yet had a change
> > to try
> > and reproduce it or do any snooping around.
> >
> > There haven't been any recent changes to readdir in the NFS client,
> > except a trivial one that adds a check for vnode type being VDIR,
> > so I don't see that it can be a recent NFS change.
> >
> > If you can increase the readdirsize, try that to see if it avoids
> > the problem. "nfsstat -m" shows you what the mount options end up
> > being after doing the mount. The server might be limiting the
> > readdirsize
> > to 4K, so you should check, even if you specify a large value for
> > the mount.
> 
> I don't know about current, but on 9.1-stable, the nfsstat -m only
> works
> for root! nfsstat can be run by anybody.
> 
Yep. I played is "safe" and only allowed root to do it.
I thought that some sysadmins might not want users to know what NFS
mounts are being done and I didn't see any need for non-root to be
able to do it.

Having said the above, I don't have a strong opinion on it or an obvious
example of a security risk caused by opening it up, so if the collective
thinks it should be doable by non-root, it can be changed.

rick

> >
> > rick
> >
> > > Regards,
> > > harti
> > > ___
> > > freebsd-current@freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to
> > > "freebsd-current-unsubscr...@freebsd.org"
> > ___
> > freebsd-current@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to
> > "freebsd-current-unsubscr...@freebsd.org"
> >
> 
> 
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Daniel Braniss]

> Hartmut Brandt wrote:
> > Hi,
> > 
> > I've updated one of my -current machines this week (previous update
> > was in
> > february). Now I see a strange effect (it seems only on NFS mounts):
> > ls or
> > even echo * will list only some files (strange enough the first files
> > from
> > the normal, alphabetically ordered list). If I change something in the
> > directory (delete a file or create a new one) for some time the
> > complete
> > listing will appear but after sime time (seconds to a minute or so)
> > again
> > only part of the files is listed.
> > 
> > A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
> > called
> > only once (returning 4096). For a full listing getdirentries is called
> > 5
> > times with the last returning 0.
> > 
> > I can still open files that are not listed if I know their name,
> > though.
> > 
> > The NFS server is a Windows 2008 server with an OpenText NFS Server
> > which
> > works without problems to all the other FreeBSD machines.
> > 
> > So what could that be?
> > 
> Someone else reported missing files returned via "ls" recently, when
> they used a small readdirsize (below 8K). I haven't yet had a change to try
> and reproduce it or do any snooping around.
> 
> There haven't been any recent changes to readdir in the NFS client,
> except a trivial one that adds a check for vnode type being VDIR,
> so I don't see that it can be a recent NFS change.
> 
> If you can increase the readdirsize, try that to see if it avoids
> the problem. "nfsstat -m" shows you what the mount options end up
> being after doing the mount. The server might be limiting the readdirsize
> to 4K, so you should check, even if you specify a large value for
> the mount.

I don't know about current, but on 9.1-stable, the nfsstat -m only works
for root! nfsstat can be run by anybody.

> 
> rick
> 
> > Regards,
> > harti
> > ___
> > freebsd-current@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to
> > "freebsd-current-unsubscr...@freebsd.org"
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
> 


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Hartmut Brandt]

On Fri, 3 May 2013, Daniel Braniss wrote:

DB>I don't know about current, but on 9.1-stable, the nfsstat -m only works
DB>for root! nfsstat can be run by anybody.

Same for current. It silently prints nothing. Took me some time
to figure out I should try as root...

harti
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Hartmut Brandt]

Hi Rick,

I checked. readdirsize is 64k.

I will try to do a binary search for the problematic commit next week. For 
this I had to do a local checkout (usually I have the system sources on 
the file server and this fails, of course).

harti

On Fri, 3 May 2013, Rick Macklem wrote:

RM>Hartmut Brandt wrote:
RM>> Hi,
RM>> 
RM>> I've updated one of my -current machines this week (previous update
RM>> was in
RM>> february). Now I see a strange effect (it seems only on NFS mounts):
RM>> ls or
RM>> even echo * will list only some files (strange enough the first files
RM>> from
RM>> the normal, alphabetically ordered list). If I change something in the
RM>> directory (delete a file or create a new one) for some time the
RM>> complete
RM>> listing will appear but after sime time (seconds to a minute or so)
RM>> again
RM>> only part of the files is listed.
RM>> 
RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
RM>> called
RM>> only once (returning 4096). For a full listing getdirentries is called
RM>> 5
RM>> times with the last returning 0.
RM>> 
RM>> I can still open files that are not listed if I know their name,
RM>> though.
RM>> 
RM>> The NFS server is a Windows 2008 server with an OpenText NFS Server
RM>> which
RM>> works without problems to all the other FreeBSD machines.
RM>> 
RM>> So what could that be?
RM>> 
RM>Someone else reported missing files returned via "ls" recently, when
RM>they used a small readdirsize (below 8K). I haven't yet had a change to try
RM>and reproduce it or do any snooping around.
RM>
RM>There haven't been any recent changes to readdir in the NFS client,
RM>except a trivial one that adds a check for vnode type being VDIR,
RM>so I don't see that it can be a recent NFS change.
RM>
RM>If you can increase the readdirsize, try that to see if it avoids
RM>the problem. "nfsstat -m" shows you what the mount options end up
RM>being after doing the mount. The server might be limiting the readdirsize
RM>to 4K, so you should check, even if you specify a large value for
RM>the mount.
RM>
RM>rick
RM>
RM>> Regards,
RM>> harti
RM>> ___
RM>> freebsd-current@freebsd.org mailing list
RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
RM>> To unsubscribe, send any mail to
RM>> "freebsd-current-unsubscr...@freebsd.org"
RM>
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> Hi,
> 
> I've updated one of my -current machines this week (previous update
> was in
> february). Now I see a strange effect (it seems only on NFS mounts):
> ls or
> even echo * will list only some files (strange enough the first files
> from
> the normal, alphabetically ordered list). If I change something in the
> directory (delete a file or create a new one) for some time the
> complete
> listing will appear but after sime time (seconds to a minute or so)
> again
> only part of the files is listed.
> 
> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
> called
> only once (returning 4096). For a full listing getdirentries is called
> 5
> times with the last returning 0.
> 
> I can still open files that are not listed if I know their name,
> though.
> 
> The NFS server is a Windows 2008 server with an OpenText NFS Server
> which
> works without problems to all the other FreeBSD machines.
> 
> So what could that be?
> 
Someone else reported missing files returned via "ls" recently, when
they used a small readdirsize (below 8K). I haven't yet had a change to try
and reproduce it or do any snooping around.

There haven't been any recent changes to readdir in the NFS client,
except a trivial one that adds a check for vnode type being VDIR,
so I don't see that it can be a recent NFS change.

If you can increase the readdirsize, try that to see if it avoids
the problem. "nfsstat -m" shows you what the mount options end up
being after doing the mount. The server might be limiting the readdirsize
to 4K, so you should check, even if you specify a large value for
the mount.

rick

> Regards,
> harti
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Hartmut Brandt]

On Thu, 2 May 2013, Freddie Cash wrote:

FC>There was just a security update that dealt with changes to getdirent or
FC>something along those lines.
FC>
FC>Check the security notices, and then see if reverting that change makes a
FC>difference.
FC>
FC>It was just in the past week here.

If you mean SA-13:05.nfsserver that seams to be related to the NFS server 
only. My problem is in the client.

harti

FC>
FC>
FC>On Thu, May 2, 2013 at 9:11 AM, Hartmut Brandt 
FC>wrote:
FC>  Hi,
FC>
FC>  I've updated one of my -current machines this week (previous
FC>  update was in february). Now I see a strange effect (it seems
FC>  only on NFS mounts): ls or even echo * will list only some files
FC>  (strange enough the first files from the normal, alphabetically
FC>  ordered list). If I change something in the directory (delete a
FC>  file or create a new one) for some time the complete listing
FC>  will appear but after sime time (seconds to a minute or so)
FC>  again only part of the files is listed.
FC>
FC>  A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
FC>  called only once (returning 4096). For a full listing
FC>  getdirentries is called 5 times with the last returning 0.
FC>
FC>  I can still open files that are not listed if I know their name,
FC>  though.
FC>
FC>  The NFS server is a Windows 2008 server with an OpenText NFS
FC>  Server which works without problems to all the other FreeBSD
FC>  machines.
FC>
FC>  So what could that be?
FC>
FC>  Regards,
FC>  harti
FC>  ___
FC>  freebsd-current@freebsd.org mailing list
FC>  http://lists.freebsd.org/mailman/listinfo/freebsd-current
FC>  To unsubscribe, send any mail to
FC>  "freebsd-current-unsubscr...@freebsd.org"
FC>
FC>
FC>
FC>
FC>--
FC>Freddie Cash
FC>fjwc...@gmail.com
FC>
FC>
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: files disappearing from ls on NFS

[Freddie Cash]

There was just a security update that dealt with changes to getdirent or
something along those lines.

Check the security notices, and then see if reverting that change makes a
difference.

It was just in the past week here.


On Thu, May 2, 2013 at 9:11 AM, Hartmut Brandt wrote:

> Hi,
>
> I've updated one of my -current machines this week (previous update was in
> february). Now I see a strange effect (it seems only on NFS mounts): ls or
> even echo * will list only some files (strange enough the first files from
> the normal, alphabetically ordered list). If I change something in the
> directory (delete a file or create a new one) for some time the complete
> listing will appear but after sime time (seconds to a minute or so) again
> only part of the files is listed.
>
> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is called
> only once (returning 4096). For a full listing getdirentries is called 5
> times with the last returning 0.
>
> I can still open files that are not listed if I know their name, though.
>
> The NFS server is a Windows 2008 server with an OpenText NFS Server which
> works without problems to all the other FreeBSD machines.
>
> So what could that be?
>
> Regards,
> harti
> __**_
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/**mailman/listinfo/freebsd-**current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@**
> freebsd.org "
>



-- 
Freddie Cash
fjwc...@gmail.com
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

files disappearing from ls on NFS

[Hartmut Brandt]

Hi,

I've updated one of my -current machines this week (previous update was in 
february). Now I see a strange effect (it seems only on NFS mounts): ls or 
even echo * will list only some files (strange enough the first files from 
the normal, alphabetically ordered list). If I change something in the 
directory (delete a file or create a new one) for some time the complete 
listing will appear but after sime time (seconds to a minute or so) again 
only part of the files is listed.


A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is called 
only once (returning 4096). For a full listing getdirentries is called 5 
times with the last returning 0.


I can still open files that are not listed if I know their name, though.

The NFS server is a Windows 2008 server with an OpenText NFS Server which 
works without problems to all the other FreeBSD machines.


So what could that be?

Regards,
harti
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"