If one uses the file offset fields in the OVERLAPPED struct and passes this to Read/WriteFile then there is no need to call SetFilePointer, and we have an atomic operation. As a result the race condition between SetFilePointer and Read/WriteFile is eliminated. Similar methods should be exposed in the framework.
On Fri, 16 Dec 2005 09:57:10 -0500, Peter Ritchie <[EMAIL PROTECTED]> wrote: >Davis, >> >>In OVERLAPPED there is an Offset and an OffsetHight member. These are not >>reserved. And your case for SetFileEx is just flat wrong. > >Sorry, I mis-read the internal-use-only for Offset and OffsetHigh. From >the documentation for SetFilePointer >http://msdn.microsoft.com/library/default.asp?url=/library/en- >us/fileio/fs/setfilepointer.asp (http://tinyurl.com/9lpvn): >"Be careful when you set a file pointer in a multi-threaded application. >You must synchronize access to shared resources. For example, an >application with threads that share a file handle, update the file >pointer, and read from the file must protect this sequence by using a >critical section object or mutex object." >Sounds pretty much the same as what I was saying (admitting my mistake >regarding the two members of OVERLAPPED). > >> >>I'm guessing you haven't done much async file programming. Using IO >>completion ports or overlapped io through this struct can dramatically >>increase throughput for an app. This is due primarily to the fact that >the >>io scheduler on the hard drive (or raid array, etc.) can actually handle >>multiple reads/writes at the same time. Having multiple outstanding io >>requests at a time is much more efficient than only doing one at a time. >> > >I've been programming IO asynchronously (not the same as multi-threaded >IO) for years (not with disk drives, through native WriteFile/ReadFile >with high-latency devices)... > >>The point is to not have your app block on a single IO operation. Instead >>of having 10 threads doing reads/writes one can use WaitForMultipleObjects >>or GetQueuedCompletion status to receive IO notifications on a single >>thread. > >That was my point, and that's what BeginRead/BeginWrite do: use WriteFile >with an overlapped structure. Asynchronous IO is different then multi- >threaded IO. I wasn't suggesting that overlapped IO should be avoided or >that it doesn't improve the responsiveness of an application. Quite the >contrary, my overlapped IO apps would be horribly unresponsive if I didn't >use overlapped IO. In the general case (regardless of whether it's IO) >adding more than a background thread to an GUI application does not >increase performance (it may decrease performance depending on the non- >.NET runtime libraries you're using--I can't comment on the .NET runtime, >in this case). I do advocate avoiding doing anything but UI logic on the >UI thread; but, that's usually just a case of adding a background thread >and/or using overlapped IO (in my overlapped IO code, I use a background >thread to perform the read/write; but, again, I'm using high-latency >devices.) > >RAID doesn't imply that the array can handle multiple accesses at the same >time; it has a queue as any other (useful) mass-storage device does. Two >threads waiting for the device's queue/cache to finalize their >asynchronous file access isn't going to be any more responsive than an a >single background thread waiting for two asynchronous file accesses. > >I guess the question is, why is asynchronous IO with FileStream.BeginWrite >()/BeginRead() not good enough for your purposes? If you want to do multi- >threaded access with FileStream you're either going to have to synchronize >access to a single FileStream object, or use one FileStream object per >thread. This is exactly how FileStream is documented; there's no bug. > >If you want a thread-safe object to perform multi-threaded asynchronous IO >on the same file handle you'll have to write your own; that's not what >FileStream is intended for. > >http://www.peterRitchie.com/ > >On Thu, 15 Dec 2005 13:29:49 -0600, John Davis <[EMAIL PROTECTED]> wrote: > >>See: >>http://msdn.microsoft.com/library/default.asp?url=/library/en- >us/dllproc/base/overlapped_str.asp >>(watch the wrap on the url) >> >>Ritchie, > > >> >>>From: Peter Ritchie <[EMAIL PROTECTED]> >>>Reply-To: "Discussion of advanced .NET topics." >>><[email protected]> >>>To: [email protected] >>>Subject: Re: [ADVANCED-DOTNET] Possible defect in BeginWrite on >FileStream >>>Date: Thu, 15 Dec 2005 12:10:18 -0500 >>> >>>BTW, you're not supposed to use the members of OVERLAPPED (other than >>>hEvent during creation). You can certainly read them, for whatever it's >>>worth; but the remaining four members are documented as "Reserved for >>>operating system use.". So, you can't get what you want with WriteFile >>>and the OVERLAPPED struct. You're supposed to use SetFilePointer[Ex] to >>>seek to a position in a file and begin writing. And, if it isn't >obvious, >>>there's only one file pointer to file: multi-threaded applications must >>>synchronize access to the file. >>> >>>What do you think multi-threaded writing to a file actually buys you? >>>There's only one head on the hard drive that can perform the write. Only >>>one thread is really going to be writing at a time anyway; even on a >multi- >>>processor computer. You might be able to get two threads to write the >>>cache at the same time with a multi-processor computer (you'd have to >have >>>at least 4 processors; the main processor is usually busy time-slicing >the >>>majority of operations); but, again, what does that do for you? >>> >>>Overlapped IO is nice because your GUI thread can be responsive while a >>>potentially long read/write operation is done "on the background"; but, >it >>>doesn't imply multi-threaded access to a file. >>> >>>http://www.peterRitchie.com/ >>> >>> >>>On Thu, 15 Dec 2005 10:01:39 -0500, John Davis <[EMAIL PROTECTED]> wrote: >>> >>> >Ian I'm afraid you're really mistaken on this one. >>> > >>> >What I'm looking for is the equivalent of a ReadFile or WriteFile call >>> >where the offset is passed in the OVERLAPPED struct. The call can >occur >>> >through overlapped IO, or an IO completion port. The call is also >>>atomic. >>> > >>> >There needs to be a random access async read/write of some sort which >>>uses >>> >the OVERLAPPED struct in the Win32 API in that it has a file offset >field >>> >for the read or write. If one observes the current implementation in >>> >DotNetReflector, the OVERLAPPED struct is getting populated via the >>> >FileStream Seek call. The OVERLAPPED struct is then used on the next >>> >async write. Why implement it this way if FileStreams can only be used >>>by >>> >a single thread? This is a bit of a cluster if you ask me. >>> > >>> >On Thu, 26 May 2005 23:22:45 +0100, Ian Griffiths <[EMAIL PROTECTED] >SW.CO.UK> >>> >wrote: >>> > >>> >>Even if there were an atomic method how would that help? It wouldn't >>> >>eliminate the race condition, because it would still be illegal to >call >>> >>the method simultaneously on multiple threads. (FileStream's >>> >>documentation says it's illegal to do that.) So you'd still need to >use >>> >>locking or some other synchronization method to make sure your threads >>> >>take it in turns even if there were an atomic method. >>> >> >>> >>I agree it would be cleaner to have a random access method if random >>> >>access is what you're doing - a stream isn't really the right >>> >>abstraction in that case. (Sadly it's the only one we're offered >>> >>AFAIK.) >>> >> >>> >>But is this a defect? It's a bug to use a stream from multiple >threads >>> >>without synchronization. It would still be a bug even with the atomic >>> >>method you propose. >>> >> >>> >> >>> >>-- >>> >>Ian Griffiths >>> >>http://www.interact-sw.co.uk/iangblog/ >>> >> >>> >>> -----Original Message----- >>> >>> From: John Davis >>> >>> >>> >>> >Are you saying that locking, Seeking, BeginWriteing, and unlocking >is >>> >>> not good enough? >>> >>> >>> >>> Well, I guess it is good enough, just doesn't seem very elegant. >>> >>Would be >>> >>> nice if there was an offset in there so one "atomic" call could be >>> >>made. >>> >>> Same as with Win32 overlapped io. >>> >>> >>> >>> >Does the BeginWrite not in effect take a snapshot of >>> >>> the seek location when the operation starts? >>> >>> >>> >>> I don't know for certain if it does, I certainly hope so. Time to >go >>> >>> digging with reflector I guess. >>> >>> >>> >>> Ian Griffiths <[EMAIL PROTECTED]> wrote: >>> >>> The documentation says that members of this class are not thread >safe, >>> >>> so multiple threads shouldn't be using this at any given instance. >You >>> >>> need to do some kind of locking if you have multiple threads using >it. >>> >>> >>> >>> Are you saying that locking, Seeking, BeginWriteing, and unlocking >is >>> >>> not good enough? Does the BeginWrite not in effect take a snapshot >of >>> >>> the seek location when the operation starts? >>> >>> >>> >>> >>> >>> > -----Original Message----- >>> >>> > From: John Davis >>> >>> > >>> >>> > The way I understand it, the current BeginWrite on the >>> >>> > FileStream object will write out it's buffer at the location of >>> >>> > the last Seek. The problem comes when there are multiple threads >>> >>> > using the same FileStream instance. >>> >>> > >>> >>> > Each one is looking to write to a different offset in the file. >This >>> >>> > means each thread must call FileStream.Seek followed by >>> >>> > FileStream.BeginWrite Obviously, this creates a race condition. >>> >>> > >>> >>> > Why isn't there an async BeginWrite, on the FileStream, with an >>> >>> > argument which specifies the offset within the file to write to. >>> >>> > It's possible to call WriteFileEx within the Win32 API with an >>> >>> > OVERLAPPED struct which specifies the offset. Why isn't this >>> >>> > exposed through the framework? >>> >> >>> >>=================================== >>> >>This list is hosted by DevelopMentor® http://www.develop.com >>> >> >>> >>View archives and manage your subscription(s) at >>> >http://discuss.develop.com >>> > >>> >There needs to be a random access async read/write of some sort which >>>uses >>> >the OVERLAPPED struct in the Win32 API in that it has a file offset >field >>> >for the read or write. If one observes the current implementation in >>> >DotNetReflector, the OVERLAPPED struct is getting populated via the >>> >FileStream Seek call. The OVERLAPPED struct is then used on the next >>> >async write. Why implement it this way if FileStreams can only be used >>>by >>> >a single thread? This is a bit of a cluster if you ask me. >>> >>>=================================== >>>This list is hosted by DevelopMentor® http://www.develop.com >>> >>>View archives and manage your subscription(s) at >http://discuss.develop.com >> >>=================================== >>This list is hosted by DevelopMentor® http://www.develop.com >> >>View archives and manage your subscription(s) at >http://discuss.develop.com > >s > >=================================== >This list is hosted by DevelopMentor® http://www.develop.com > >View archives and manage your subscription(s) at http://discuss.develop.com =================================== This list is hosted by DevelopMentor® http://www.develop.com View archives and manage your subscription(s) at http://discuss.develop.com
