Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-26 Thread Robert Love
On Fri, 2005-08-26 at 13:52 -0400, John McCutchan wrote: > Thanks for your suggestion, it has fixed the inotify problem. But where > to put the fix is turning into a bit of a mess. Some callers like > drivers/md/dm.c:682 call idr_get_new_above as if it will return >= > starting_id. The comment

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-26 Thread John McCutchan
On Fri, 2005-08-26 at 13:03 -0400, Jim Houston wrote: > Hi Everyone, > > I'm answering this from my home email. I have not heard from my > co-workers in Florida yet, and I imagine that they are busy cleaning up > after hurricane Katrina and waiting for the power to come back on. > > It looks

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-26 Thread Jim Houston
Hi Everyone, I'm answering this from my home email. I have not heard from my co-workers in Florida yet, and I imagine that they are busy cleaning up after hurricane Katrina and waiting for the power to come back on. It looks like we have an "off by one" problem with idr_get_new_above() which may

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-26 Thread Jim Houston
Hi Everyone, I'm answering this from my home email. I have not heard from my co-workers in Florida yet, and I imagine that they are busy cleaning up after hurricane Katrina and waiting for the power to come back on. It looks like we have an off by one problem with idr_get_new_above() which may

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-26 Thread John McCutchan
On Fri, 2005-08-26 at 13:03 -0400, Jim Houston wrote: Hi Everyone, I'm answering this from my home email. I have not heard from my co-workers in Florida yet, and I imagine that they are busy cleaning up after hurricane Katrina and waiting for the power to come back on. It looks like we

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-26 Thread Robert Love
On Fri, 2005-08-26 at 13:52 -0400, John McCutchan wrote: Thanks for your suggestion, it has fixed the inotify problem. But where to put the fix is turning into a bit of a mess. Some callers like drivers/md/dm.c:682 call idr_get_new_above as if it will return = starting_id. The comment says

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 16:10 -0700, George Anzinger wrote: > That IS strange. 1024 is on a "level" boundry, but then next level is > 2**15, not 2**11. I will take a look. Remember that the level is never filled, so maybe the smallest level just gets an offset or something? Well, you're the

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread George Anzinger
John McCutchan wrote: On Thu, 2005-08-25 at 11:54 -0700, George Anzinger wrote: Robert Love wrote: On Thu, 2005-08-25 at 09:33 -0400, John McCutchan wrote: On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: ~ I think the best thing is to take idr into user space and emulate the

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 11:54 -0700, George Anzinger wrote: > I think the best thing is to take idr into user space and emulate the > problem usage. Good plan, I guess. Do you think that's easy? > To this end, from the log it appears that you _might_ be > moving between 0, 1 and 2 entries

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 21:03 +0200, Johannes Berg wrote: > On Thu, 2005-08-25 at 11:54 -0700, George Anzinger wrote: > > > I think the best thing is to take idr into user space and emulate the > > problem usage. > > Good plan, I guess. Do you think that's easy? > > > To this end, from the log

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 11:54 -0700, George Anzinger wrote: > Robert Love wrote: > > On Thu, 2005-08-25 at 09:33 -0400, John McCutchan wrote: > > > >>On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: > >> > ~ > >>>dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1022 from >

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread George Anzinger
Robert Love wrote: On Thu, 2005-08-25 at 09:33 -0400, John McCutchan wrote: On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: ~ dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4 dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): inotify_add_watch

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Robert Love
On Thu, 2005-08-25 at 09:33 -0400, John McCutchan wrote: > On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: > > Hi, > > > > I have also observed another problem with inotify with dovecot - so I spoke > > with Johannes Berg who wrote the inotify code in dovecot. He suggested I > > post

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 16:41 +0200, Johannes Berg wrote: > On Thu, 2005-08-25 at 10:13 -0400, John McCutchan wrote: > > > I really don't want 2.6.13 to go out with this bug or the compromise. If > > we use 0, we will have a lot of wd re-use. Which will cause "strange" > > problems in inotify using

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 10:13 -0400, John McCutchan wrote: > I really don't want 2.6.13 to go out with this bug or the compromise. If > we use 0, we will have a lot of wd re-use. Which will cause "strange" > problems in inotify using applications that cleanup upon receipt of an > IN_IGNORE event.

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 16:13 +0200, Johannes Berg wrote: > On Thu, 2005-08-25 at 10:06 -0400, John McCutchan wrote: > > > it fails on 2.6.13-rc6 as soon as the device is full and doesn't hold > > > any more directories. > > Obviously this wasn't true, I was hitting the 8192 watches limit and >

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 10:06 -0400, John McCutchan wrote: > > it fails on 2.6.13-rc6 as soon as the device is full and doesn't hold > > any more directories. Obviously this wasn't true, I was hitting the 8192 watches limit and misinterpreted the error message. I just tested up to 10 watches

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 09:47 -0400, Robert Love wrote: > On Thu, 2005-08-25 at 09:40 -0400, John McCutchan wrote: > > > I get that message a lot. I know I have said this before (and was wrong) > > but I think the idr layer is busted. > > This time I think I agree with you. ;-) > > Let's just

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 16:03 +0200, Johannes Berg wrote: > On Thu, 2005-08-25 at 09:47 -0400, Robert Love wrote: > > > Let's just pass zero for the "above" parameter in idr_get_new_above(), > > which is I believe the behavior of the other interface, and see if the > > 1024-multiple problem goes

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 09:47 -0400, Robert Love wrote: > Let's just pass zero for the "above" parameter in idr_get_new_above(), > which is I believe the behavior of the other interface, and see if the > 1024-multiple problem goes away. We definitely did not have that > before. Will we then need

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 15:50 +0200, Johannes Berg wrote: > On Thu, 2005-08-25 at 09:40 -0400, John McCutchan wrote: > > > On 2.6.13-rc7 the test program fails. It always fails when a wd == 1024. > > If I skip inotify_rm_watch when wd == 1024, it will fail at wd == 2048. > > It seems the idr layer

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 09:40 -0400, John McCutchan wrote: > On 2.6.13-rc7 the test program fails. It always fails when a wd == 1024. > If I skip inotify_rm_watch when wd == 1024, it will fail at wd == 2048. > It seems the idr layer has an aversion to multiples of 1024. > > When I run your test

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Robert Love
On Thu, 2005-08-25 at 09:40 -0400, John McCutchan wrote: > I get that message a lot. I know I have said this before (and was wrong) > but I think the idr layer is busted. This time I think I agree with you. ;-) Let's just pass zero for the "above" parameter in idr_get_new_above(), which is I

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 14:18 +0200, Johannes Berg wrote: > Hi, > > > I have also observed another problem with inotify with dovecot - so I spoke > > with Johannes Berg who wrote the inotify code in dovecot. He suggested I > > post > > here to LKML since his opinion is that this to be a kernel

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: > Hi, > > I have also observed another problem with inotify with dovecot - so I spoke > with Johannes Berg who wrote the inotify code in dovecot. He suggested I > post > here to LKML since his opinion is that this to be a kernel bug. >

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
Hi, > I have also observed another problem with inotify with dovecot - so I spoke > with Johannes Berg who wrote the inotify code in dovecot. He suggested I > post > here to LKML since his opinion is that this to be a kernel bug. Allow me to jump in at this point. The small tool below

Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Reuben Farrelly
Hi, On 22/08/2005 9:10 p.m., John McCutchan wrote: On Sat, 2005-08-20 at 23:52 -0700, Andrew Morton wrote: Reuben Farrelly <[EMAIL PROTECTED]> wrote: Hi, On 19/08/2005 11:37 a.m., Andrew Morton wrote: ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc6/2.6.13-rc6-mm1/

Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Reuben Farrelly
Hi, On 22/08/2005 9:10 p.m., John McCutchan wrote: On Sat, 2005-08-20 at 23:52 -0700, Andrew Morton wrote: Reuben Farrelly [EMAIL PROTECTED] wrote: Hi, On 19/08/2005 11:37 a.m., Andrew Morton wrote: ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc6/2.6.13-rc6-mm1/ -

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
Hi, I have also observed another problem with inotify with dovecot - so I spoke with Johannes Berg who wrote the inotify code in dovecot. He suggested I post here to LKML since his opinion is that this to be a kernel bug. Allow me to jump in at this point. The small tool below triggers

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: Hi, I have also observed another problem with inotify with dovecot - so I spoke with Johannes Berg who wrote the inotify code in dovecot. He suggested I post here to LKML since his opinion is that this to be a kernel bug. The

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 14:18 +0200, Johannes Berg wrote: Hi, I have also observed another problem with inotify with dovecot - so I spoke with Johannes Berg who wrote the inotify code in dovecot. He suggested I post here to LKML since his opinion is that this to be a kernel bug.

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Robert Love
On Thu, 2005-08-25 at 09:40 -0400, John McCutchan wrote: I get that message a lot. I know I have said this before (and was wrong) but I think the idr layer is busted. This time I think I agree with you. ;-) Let's just pass zero for the above parameter in idr_get_new_above(), which is I

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 09:40 -0400, John McCutchan wrote: On 2.6.13-rc7 the test program fails. It always fails when a wd == 1024. If I skip inotify_rm_watch when wd == 1024, it will fail at wd == 2048. It seems the idr layer has an aversion to multiples of 1024. When I run your test program

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 09:47 -0400, Robert Love wrote: Let's just pass zero for the above parameter in idr_get_new_above(), which is I believe the behavior of the other interface, and see if the 1024-multiple problem goes away. We definitely did not have that before. Will we then need to

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 15:50 +0200, Johannes Berg wrote: On Thu, 2005-08-25 at 09:40 -0400, John McCutchan wrote: On 2.6.13-rc7 the test program fails. It always fails when a wd == 1024. If I skip inotify_rm_watch when wd == 1024, it will fail at wd == 2048. It seems the idr layer has an

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 16:03 +0200, Johannes Berg wrote: On Thu, 2005-08-25 at 09:47 -0400, Robert Love wrote: Let's just pass zero for the above parameter in idr_get_new_above(), which is I believe the behavior of the other interface, and see if the 1024-multiple problem goes away. We

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 09:47 -0400, Robert Love wrote: On Thu, 2005-08-25 at 09:40 -0400, John McCutchan wrote: I get that message a lot. I know I have said this before (and was wrong) but I think the idr layer is busted. This time I think I agree with you. ;-) Let's just pass zero

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 10:06 -0400, John McCutchan wrote: it fails on 2.6.13-rc6 as soon as the device is full and doesn't hold any more directories. Obviously this wasn't true, I was hitting the 8192 watches limit and misinterpreted the error message. I just tested up to 10 watches with

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 16:13 +0200, Johannes Berg wrote: On Thu, 2005-08-25 at 10:06 -0400, John McCutchan wrote: it fails on 2.6.13-rc6 as soon as the device is full and doesn't hold any more directories. Obviously this wasn't true, I was hitting the 8192 watches limit and

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 10:13 -0400, John McCutchan wrote: I really don't want 2.6.13 to go out with this bug or the compromise. If we use 0, we will have a lot of wd re-use. Which will cause strange problems in inotify using applications that cleanup upon receipt of an IN_IGNORE event. What

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 16:41 +0200, Johannes Berg wrote: On Thu, 2005-08-25 at 10:13 -0400, John McCutchan wrote: I really don't want 2.6.13 to go out with this bug or the compromise. If we use 0, we will have a lot of wd re-use. Which will cause strange problems in inotify using

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Robert Love
On Thu, 2005-08-25 at 09:33 -0400, John McCutchan wrote: On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: Hi, I have also observed another problem with inotify with dovecot - so I spoke with Johannes Berg who wrote the inotify code in dovecot. He suggested I post here to

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread George Anzinger
Robert Love wrote: On Thu, 2005-08-25 at 09:33 -0400, John McCutchan wrote: On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: ~ dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4 dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): inotify_add_watch

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 11:54 -0700, George Anzinger wrote: Robert Love wrote: On Thu, 2005-08-25 at 09:33 -0400, John McCutchan wrote: On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: ~ dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4 dovecot:

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread John McCutchan
On Thu, 2005-08-25 at 21:03 +0200, Johannes Berg wrote: On Thu, 2005-08-25 at 11:54 -0700, George Anzinger wrote: I think the best thing is to take idr into user space and emulate the problem usage. Good plan, I guess. Do you think that's easy? To this end, from the log it appears

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 11:54 -0700, George Anzinger wrote: I think the best thing is to take idr into user space and emulate the problem usage. Good plan, I guess. Do you think that's easy? To this end, from the log it appears that you _might_ be moving between 0, 1 and 2 entries

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread George Anzinger
John McCutchan wrote: On Thu, 2005-08-25 at 11:54 -0700, George Anzinger wrote: Robert Love wrote: On Thu, 2005-08-25 at 09:33 -0400, John McCutchan wrote: On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote: ~ I think the best thing is to take idr into user space and emulate the

Re: Inotify problem [was Re: 2.6.13-rc6-mm1]

2005-08-25 Thread Johannes Berg
On Thu, 2005-08-25 at 16:10 -0700, George Anzinger wrote: That IS strange. 1024 is on a level boundry, but then next level is 2**15, not 2**11. I will take a look. Remember that the level is never filled, so maybe the smallest level just gets an offset or something? Well, you're the expert