Re: sys_write() racy for multi-threaded append?

2007-03-16 Thread Frank Ch. Eigler
"Michael K. Edwards" <[EMAIL PROTECTED]> writes: > Thanks Alan, this is really helpful -- I'll see if I can figure out > kprobes. It is not immediately obvious to me how to use them to > trace function calls in userspace, but I'll read up. kprobes does not work with userspace. We in systemtap

Re: sys_write() racy for multi-threaded append?

2007-03-16 Thread Alan Cox
> > gdbstubs is also not terribly SMP aware and for low level work its > > sometimes easier to have on gdb per processor if you can get your brain > > around it. > > That's a trick I don't know. What do you do, fire up the target > process, put the whole thing to sleep with a SIGSTOP, and then

Re: sys_write() racy for multi-threaded append?

2007-03-16 Thread Alan Cox
gdbstubs is also not terribly SMP aware and for low level work its sometimes easier to have on gdb per processor if you can get your brain around it. That's a trick I don't know. What do you do, fire up the target process, put the whole thing to sleep with a SIGSTOP, and then attach a

Re: sys_write() racy for multi-threaded append?

2007-03-16 Thread Frank Ch. Eigler
Michael K. Edwards [EMAIL PROTECTED] writes: Thanks Alan, this is really helpful -- I'll see if I can figure out kprobes. It is not immediately obvious to me how to use them to trace function calls in userspace, but I'll read up. kprobes does not work with userspace. We in systemtap land

Re: sys_write() racy for multi-threaded append?

2007-03-14 Thread Michael K. Edwards
Thanks Alan, this is really helpful -- I'll see if I can figure out kprobes. It is not immediately obvious to me how to use them to trace function calls in userspace, but I'll read up. On 3/13/07, Alan Cox <[EMAIL PROTECTED]> wrote: > But on that note -- do you have any idea how one might get

Re: sys_write() racy for multi-threaded append?

2007-03-14 Thread Michael K. Edwards
Thanks Alan, this is really helpful -- I'll see if I can figure out kprobes. It is not immediately obvious to me how to use them to trace function calls in userspace, but I'll read up. On 3/13/07, Alan Cox [EMAIL PROTECTED] wrote: But on that note -- do you have any idea how one might get

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
In case anyone cares, this is a snippet of my work-in-progress read_write.c illustrating how I might handle f_pos. Can anyone point me to data showing whether it's worth avoiding the spinlock when the "struct file" is not shared between threads? (In my world, correctness comes before

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
On 3/13/07, Christoph Hellwig <[EMAIL PROTECTED]> wrote: Michael, please stop spreading this utter bullshit _now_. You're so full of half-knowledge that it's not funny anymore, and you try to insult people knowing a few magniutes more than you left and right. Thank you Christoph for that

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Christoph Hellwig
Michael, please stop spreading this utter bullshit _now_. You're so full of half-knowledge that it's not funny anymore, and you try to insult people knowing a few magniutes more than you left and right. Can you please select a different mailinglist for your trolling? - To unsubscribe from this

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
Clearly f_pos atomicity has been handled differently in the not-so-distant past: http://www.mail-archive.com/linux-fsdevel@vger.kernel.org/msg01628.html And equally clearly the current generic_file_llseek semantics are erroneous for large offsets, and we shouldn't be taking the inode mutex in

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
On 3/13/07, David Miller <[EMAIL PROTECTED]> wrote: You're not even safe over standard output, simply run the program over ssh and you suddenly have socket semantics to deal with. I'm intimately familiar with this one. Easily worked around by piping the output through cat or tee. Not that

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread David M. Lloyd
On Tue, 2007-03-13 at 02:24 +, Alan Cox wrote: > > purports to handle short writes but has never been exercised is > > arguably worse than code that simply bombs on short write. So if I > > can't shim in an induce-short-writes-randomly-on-purpose mechanism > > during development, I don't want

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Alan Cox
> But on that note -- do you have any idea how one might get ltrace to > work on a multi-threaded program, or how one might enhance it to > instrument function calls from one shared library to another? Or I don't know a vast amount about ARM ELF user space so no. > better yet, can you advise me

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread David Miller
From: "Michael K. Edwards" <[EMAIL PROTECTED]> Date: Mon, 12 Mar 2007 23:25:48 -0800 > Quality means the devices you ship now keep working in the field, and > the probable cost of later rework if the requirements change does not > exceed the opportunity cost of over-engineering up front. Economy

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
On 3/12/07, Alan Cox <[EMAIL PROTECTED]> wrote: > Writing to a file from multiple processes is not usually the problem. > Writing to a common "struct file" from multiple threads is. Not normally because POSIX sensibly invented pread/pwrite. Forgot preadv/pwritev but they did the basics and end

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
On 3/12/07, Alan Cox [EMAIL PROTECTED] wrote: Writing to a file from multiple processes is not usually the problem. Writing to a common struct file from multiple threads is. Not normally because POSIX sensibly invented pread/pwrite. Forgot preadv/pwritev but they did the basics and end of

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread David Miller
From: Michael K. Edwards [EMAIL PROTECTED] Date: Mon, 12 Mar 2007 23:25:48 -0800 Quality means the devices you ship now keep working in the field, and the probable cost of later rework if the requirements change does not exceed the opportunity cost of over-engineering up front. Economy gets

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Alan Cox
But on that note -- do you have any idea how one might get ltrace to work on a multi-threaded program, or how one might enhance it to instrument function calls from one shared library to another? Or I don't know a vast amount about ARM ELF user space so no. better yet, can you advise me on

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread David M. Lloyd
On Tue, 2007-03-13 at 02:24 +, Alan Cox wrote: purports to handle short writes but has never been exercised is arguably worse than code that simply bombs on short write. So if I can't shim in an induce-short-writes-randomly-on-purpose mechanism during development, I don't want short

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
On 3/13/07, David Miller [EMAIL PROTECTED] wrote: You're not even safe over standard output, simply run the program over ssh and you suddenly have socket semantics to deal with. I'm intimately familiar with this one. Easily worked around by piping the output through cat or tee. Not that one

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
Clearly f_pos atomicity has been handled differently in the not-so-distant past: http://www.mail-archive.com/linux-fsdevel@vger.kernel.org/msg01628.html And equally clearly the current generic_file_llseek semantics are erroneous for large offsets, and we shouldn't be taking the inode mutex in

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Christoph Hellwig
Michael, please stop spreading this utter bullshit _now_. You're so full of half-knowledge that it's not funny anymore, and you try to insult people knowing a few magniutes more than you left and right. Can you please select a different mailinglist for your trolling? - To unsubscribe from this

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
On 3/13/07, Christoph Hellwig [EMAIL PROTECTED] wrote: Michael, please stop spreading this utter bullshit _now_. You're so full of half-knowledge that it's not funny anymore, and you try to insult people knowing a few magniutes more than you left and right. Thank you Christoph for that

Re: sys_write() racy for multi-threaded append?

2007-03-13 Thread Michael K. Edwards
In case anyone cares, this is a snippet of my work-in-progress read_write.c illustrating how I might handle f_pos. Can anyone point me to data showing whether it's worth avoiding the spinlock when the struct file is not shared between threads? (In my world, correctness comes before code-bumming

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Alan Cox
> Writing to a file from multiple processes is not usually the problem. > Writing to a common "struct file" from multiple threads is. Not normally because POSIX sensibly invented pread/pwrite. Forgot preadv/pwritev but they did the basics and end of problem > So what? My products are shipping

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Michael K. Edwards
On 3/12/07, Bodo Eggert <[EMAIL PROTECTED]> wrote: On Mon, 12 Mar 2007, Michael K. Edwards wrote: > That's fine when you're doing integration test, and should probably be > the default during development. But if the race is first exposed in > the field, or if the developer is trying to

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Bodo Eggert
On Mon, 12 Mar 2007, Michael K. Edwards wrote: > On 3/12/07, Bodo Eggert <[EMAIL PROTECTED]> wrote: > > Michael K. Edwards <[EMAIL PROTECTED]> wrote: > > > Actually, I think it would make the kernel (negligibly) faster to bump > > > f_pos before the vfs_write() call. > > > > This is a security

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Michael K. Edwards
On 3/12/07, Bodo Eggert <[EMAIL PROTECTED]> wrote: Michael K. Edwards <[EMAIL PROTECTED]> wrote: > Actually, I think it would make the kernel (negligibly) faster to bump > f_pos before the vfs_write() call. This is a security risk. other process: unlink(secrest_file) Thread

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Bodo Eggert
Michael K. Edwards <[EMAIL PROTECTED]> wrote: > On 3/8/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: >> Absolutely not. We dont want to slow down kernel 'just in case a fool might >> want to do crazy things' > > Actually, I think it would make the kernel (negligibly) faster to bump > f_pos before

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Bodo Eggert
Michael K. Edwards [EMAIL PROTECTED] wrote: On 3/8/07, Eric Dumazet [EMAIL PROTECTED] wrote: Absolutely not. We dont want to slow down kernel 'just in case a fool might want to do crazy things' Actually, I think it would make the kernel (negligibly) faster to bump f_pos before the

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Michael K. Edwards
On 3/12/07, Bodo Eggert [EMAIL PROTECTED] wrote: Michael K. Edwards [EMAIL PROTECTED] wrote: Actually, I think it would make the kernel (negligibly) faster to bump f_pos before the vfs_write() call. This is a security risk. other process: unlink(secrest_file) Thread 1:

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Bodo Eggert
On Mon, 12 Mar 2007, Michael K. Edwards wrote: On 3/12/07, Bodo Eggert [EMAIL PROTECTED] wrote: Michael K. Edwards [EMAIL PROTECTED] wrote: Actually, I think it would make the kernel (negligibly) faster to bump f_pos before the vfs_write() call. This is a security risk. snip I

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Michael K. Edwards
On 3/12/07, Bodo Eggert [EMAIL PROTECTED] wrote: On Mon, 12 Mar 2007, Michael K. Edwards wrote: That's fine when you're doing integration test, and should probably be the default during development. But if the race is first exposed in the field, or if the developer is trying to concentrate

Re: sys_write() racy for multi-threaded append?

2007-03-12 Thread Alan Cox
Writing to a file from multiple processes is not usually the problem. Writing to a common struct file from multiple threads is. Not normally because POSIX sensibly invented pread/pwrite. Forgot preadv/pwritev but they did the basics and end of problem So what? My products are shipping _now_.

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Michael K. Edwards
I apologize for throwing around words like "stupid". Whether or not the current semantics can be improved, that's not a constructive way to characterize them. I'm sorry. As three people have ably pointed out :-), the particular case of a pipe/FIFO isn't seekable and doesn't need the f_pos

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Benjamin LaHaise
On Fri, Mar 09, 2007 at 04:19:55AM -0800, Michael K. Edwards wrote: > On 3/8/07, Benjamin LaHaise <[EMAIL PROTECTED]> wrote: > >Any number of things can cause a short write to occur, and rewinding the > >file position after the fact is just as bad. A sane app has to either > >serialise the writes

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Eric Dumazet
On Friday 09 March 2007 13:19, Michael K. Edwards wrote: > On 3/8/07, Benjamin LaHaise <[EMAIL PROTECTED]> wrote: > > Any number of things can cause a short write to occur, and rewinding the > > file position after the fact is just as bad. A sane app has to either > > serialise the writes itself

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Alan Cox
> 1003.1 unless O_NONBLOCK is set. (Not that f_pos is interesting on a > pipe except as a "bytes sent" indicator -- and in the multi-threaded f_pos is undefined on a FIFO or similar object. > As to what a "sane app" has to do: it's just not that unusual to write > application code that treats

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Michael K. Edwards
On 3/8/07, Benjamin LaHaise <[EMAIL PROTECTED]> wrote: Any number of things can cause a short write to occur, and rewinding the file position after the fact is just as bad. A sane app has to either serialise the writes itself or use a thread safe API like pwrite(). Not on a pipe/FIFO. Short

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Michael K. Edwards
On 3/8/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: Dont even try, you *cannot* do that, without breaking the standards, or without a performance drop. The only safe way would be to lock the file during the whole read()/write() syscall, and we dont want this (this would be more expensive than

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Michael K. Edwards
On 3/8/07, Eric Dumazet [EMAIL PROTECTED] wrote: Dont even try, you *cannot* do that, without breaking the standards, or without a performance drop. The only safe way would be to lock the file during the whole read()/write() syscall, and we dont want this (this would be more expensive than

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Michael K. Edwards
On 3/8/07, Benjamin LaHaise [EMAIL PROTECTED] wrote: Any number of things can cause a short write to occur, and rewinding the file position after the fact is just as bad. A sane app has to either serialise the writes itself or use a thread safe API like pwrite(). Not on a pipe/FIFO. Short

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Alan Cox
1003.1 unless O_NONBLOCK is set. (Not that f_pos is interesting on a pipe except as a bytes sent indicator -- and in the multi-threaded f_pos is undefined on a FIFO or similar object. As to what a sane app has to do: it's just not that unusual to write application code that treats a short

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Eric Dumazet
On Friday 09 March 2007 13:19, Michael K. Edwards wrote: On 3/8/07, Benjamin LaHaise [EMAIL PROTECTED] wrote: Any number of things can cause a short write to occur, and rewinding the file position after the fact is just as bad. A sane app has to either serialise the writes itself or use a

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Benjamin LaHaise
On Fri, Mar 09, 2007 at 04:19:55AM -0800, Michael K. Edwards wrote: On 3/8/07, Benjamin LaHaise [EMAIL PROTECTED] wrote: Any number of things can cause a short write to occur, and rewinding the file position after the fact is just as bad. A sane app has to either serialise the writes itself

Re: sys_write() racy for multi-threaded append?

2007-03-09 Thread Michael K. Edwards
I apologize for throwing around words like stupid. Whether or not the current semantics can be improved, that's not a constructive way to characterize them. I'm sorry. As three people have ably pointed out :-), the particular case of a pipe/FIFO isn't seekable and doesn't need the f_pos member

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Eric Dumazet
Michael K. Edwards a écrit : On 3/8/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: Absolutely not. We dont want to slow down kernel 'just in case a fool might want to do crazy things' Actually, I think it would make the kernel (negligibly) faster to bump f_pos before the vfs_write() call.

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Benjamin LaHaise
> Actually, I think it would make the kernel (negligibly) faster to bump > f_pos before the vfs_write() call. Unless fget_light sets fput_needed > or the write doesn't complete cleanly, you won't have to touch the > file table entry again after vfs_write() returns. You can adjust Any number of

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Michael K. Edwards
On 3/8/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: Absolutely not. We dont want to slow down kernel 'just in case a fool might want to do crazy things' Actually, I think it would make the kernel (negligibly) faster to bump f_pos before the vfs_write() call. Unless fget_light sets fput_needed

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Eric Dumazet
Michael K. Edwards a écrit : On 3/8/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: Nothing in the manuals says that write() on same fd should be non racy : In particular file pos might be undefined. There is a reason pwrite() exists. Kernel doesnt have to enforce thread safety as standard is

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Michael K. Edwards
On 3/8/07, Eric Dumazet <[EMAIL PROTECTED]> wrote: Nothing in the manuals says that write() on same fd should be non racy : In particular file pos might be undefined. There is a reason pwrite() exists. Kernel doesnt have to enforce thread safety as standard is quite clear. I know the standard

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Eric Dumazet
Michael K. Edwards a écrit : from sys_write(): file = fget_light(fd, _needed); if (file) { loff_t pos = file_pos_read(file); ret = vfs_write(file, buf, count, ); file_pos_write(file, pos); fput_light(file, fput_needed);

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Alan Cox
> Surely that's racy when two threads write to the same fd and the first > vfs_write call blocks or is preempted. Or does fget_light take some > per-file lock that I'm not seeing? I think you are making assumptions about file position behaviour that are simply not guaranteed in the standards

sys_write() racy for multi-threaded append?

2007-03-08 Thread Michael K. Edwards
from sys_write(): file = fget_light(fd, _needed); if (file) { loff_t pos = file_pos_read(file); ret = vfs_write(file, buf, count, ); file_pos_write(file, pos); fput_light(file, fput_needed); } Surely that's racy

sys_write() racy for multi-threaded append?

2007-03-08 Thread Michael K. Edwards
from sys_write(): file = fget_light(fd, fput_needed); if (file) { loff_t pos = file_pos_read(file); ret = vfs_write(file, buf, count, pos); file_pos_write(file, pos); fput_light(file, fput_needed); } Surely that's

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Alan Cox
Surely that's racy when two threads write to the same fd and the first vfs_write call blocks or is preempted. Or does fget_light take some per-file lock that I'm not seeing? I think you are making assumptions about file position behaviour that are simply not guaranteed in the standards

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Eric Dumazet
Michael K. Edwards a écrit : from sys_write(): file = fget_light(fd, fput_needed); if (file) { loff_t pos = file_pos_read(file); ret = vfs_write(file, buf, count, pos); file_pos_write(file, pos); fput_light(file,

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Michael K. Edwards
On 3/8/07, Eric Dumazet [EMAIL PROTECTED] wrote: Nothing in the manuals says that write() on same fd should be non racy : In particular file pos might be undefined. There is a reason pwrite() exists. Kernel doesnt have to enforce thread safety as standard is quite clear. I know the standard

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Eric Dumazet
Michael K. Edwards a écrit : On 3/8/07, Eric Dumazet [EMAIL PROTECTED] wrote: Nothing in the manuals says that write() on same fd should be non racy : In particular file pos might be undefined. There is a reason pwrite() exists. Kernel doesnt have to enforce thread safety as standard is

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Michael K. Edwards
On 3/8/07, Eric Dumazet [EMAIL PROTECTED] wrote: Absolutely not. We dont want to slow down kernel 'just in case a fool might want to do crazy things' Actually, I think it would make the kernel (negligibly) faster to bump f_pos before the vfs_write() call. Unless fget_light sets fput_needed or

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Benjamin LaHaise
Actually, I think it would make the kernel (negligibly) faster to bump f_pos before the vfs_write() call. Unless fget_light sets fput_needed or the write doesn't complete cleanly, you won't have to touch the file table entry again after vfs_write() returns. You can adjust Any number of

Re: sys_write() racy for multi-threaded append?

2007-03-08 Thread Eric Dumazet
Michael K. Edwards a écrit : On 3/8/07, Eric Dumazet [EMAIL PROTECTED] wrote: Absolutely not. We dont want to slow down kernel 'just in case a fool might want to do crazy things' Actually, I think it would make the kernel (negligibly) faster to bump f_pos before the vfs_write() call. Unless